1
|
Vennemeyer S, Kinnear B, Gao A, Zhu S, Nattam A, Knopp MI, Warm E, Wu DT. User-Centered Evaluation and Design Recommendations for an Internal Medicine Resident Competency Assessment Dashboard. Appl Clin Inform 2023; 14:996-1007. [PMID: 38122817 PMCID: PMC10733060 DOI: 10.1055/s-0043-1777103] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2023] [Accepted: 10/25/2023] [Indexed: 12/23/2023] Open
Abstract
OBJECTIVES Clinical Competency Committee (CCC) members employ varied approaches to the review process. This makes the design of a competency assessment dashboard that fits the needs of all members difficult. This work details a user-centered evaluation of a dashboard currently utilized by the Internal Medicine Clinical Competency Committee (IM CCC) at the University of Cincinnati College of Medicine and generated design recommendations. METHODS Eleven members of the IM CCC participated in semistructured interviews with the research team. These interviews were recorded and transcribed for analysis. The three design research methods used in this study included process mapping (workflow diagrams), affinity diagramming, and a ranking experiment. RESULTS Through affinity diagramming, the research team identified and organized opportunities for improvement about the current system expressed by study participants. These areas include a time-consuming preprocessing step, lack of integration of data from multiple sources, and different workflows for each step in the review process. Finally, the research team categorized nine dashboard components based on rankings provided by the participants. CONCLUSION We successfully conducted user-centered evaluation of an IM CCC dashboard and generated four recommendations. Programs should integrate quantitative and qualitative feedback, create multiple views to display these data based on user roles, work with designers to create a usable, interpretable dashboard, and develop a strong informatics pipeline to manage the system. To our knowledge, this type of user-centered evaluation has rarely been attempted in the medical education domain. Therefore, this study provides best practices for other residency programs to evaluate current competency assessment tools and to develop new ones.
Collapse
Affiliation(s)
- Scott Vennemeyer
- Department of Biomedical Informatics, College of Medicine, University of Cincinnati, Ohio, United States
| | - Benjamin Kinnear
- Department of Pediatrics, College of Medicine, University of Cincinnati, Ohio, United States
- Department of Internal Medicine, College of Medicine, University of Cincinnati, Ohio, United States
| | - Andy Gao
- Department of Biomedical Informatics, College of Medicine, University of Cincinnati, Ohio, United States
- Medical Sciences Baccalaureate Program, College of Medicine, University of Cincinnati, Ohio, United States
| | - Siyi Zhu
- Department of Biomedical Informatics, College of Medicine, University of Cincinnati, Ohio, United States
- School of Design, College of Design, Architecture, Art, and Planning (DAAP), University of Cincinnati, Ohio, United States
| | - Anunita Nattam
- Department of Biomedical Informatics, College of Medicine, University of Cincinnati, Ohio, United States
- Medical Sciences Baccalaureate Program, College of Medicine, University of Cincinnati, Ohio, United States
| | - Michelle I. Knopp
- Department of Internal Medicine, College of Medicine, University of Cincinnati, Ohio, United States
- Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center, Department of Pediatrics, College of Medicine, University of Cincinnati, Ohio, United States
| | - Eric Warm
- Department of Internal Medicine, College of Medicine, University of Cincinnati, Ohio, United States
| | - Danny T.Y. Wu
- Department of Biomedical Informatics, College of Medicine, University of Cincinnati, Ohio, United States
- Department of Pediatrics, College of Medicine, University of Cincinnati, Ohio, United States
- Medical Sciences Baccalaureate Program, College of Medicine, University of Cincinnati, Ohio, United States
- School of Design, College of Design, Architecture, Art, and Planning (DAAP), University of Cincinnati, Ohio, United States
| |
Collapse
|
2
|
Kinnear B, Weber DE, Schumacher DJ, Edje L, Warm EJ, Anderson HL. Reconstructing Neurath's Ship: A Case Study in Reevaluating Equity in a Program of Assessment. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:S50-S56. [PMID: 37071695 DOI: 10.1097/acm.0000000000005249] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Inequity in assessment has been described as a "wicked problem"-an issue with complex roots, inherent tensions, and unclear solutions. To address inequity, health professions educators must critically examine their implicit understandings of truth and knowledge (i.e., their epistemologies) with regard to educational assessment before jumping to solutions. The authors use the analogy of a ship (program of assessment) sailing on different seas (epistemologies) to describe their journey in seeking to improve equity in assessment. Should the education community repair the ship of assessment while sailing or should the ship be scrapped and built anew? The authors share a case study of a well-developed internal medicine residency program of assessment and describe efforts to evaluate and enable equity using various epistemological lenses. They first used a postpositivist lens to evaluate if the systems and strategies aligned with best practices, but found they did not capture important nuances of what equitable assessment entails. Next, they used a constructivist approach to improve stakeholder engagement, but found they still failed to question the inequitable assumptions inherent to their systems and strategies. Finally, they describe a shift to critical epistemologies, seeking to understand who experiences inequity and harm to dismantle inequitable systems and create better ones. The authors describe how each unique sea promoted different adaptations to their ship, and challenge programs to sail through new epistemological waters as a starting point for making their own ships more equitable.
Collapse
Affiliation(s)
- Benjamin Kinnear
- B. Kinnear is associate professor of internal medicine and pediatrics, Departments of Pediatrics and Internal Medicine, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0003-0052-4130
| | - Danielle E Weber
- D.E. Weber is assistant professor of internal medicine and pediatrics, Departments of Pediatrics and Internal Medicine, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0002-4857-6936
| | - Daniel J Schumacher
- D.J. Schumacher is tenured professor of pediatrics, Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0001-5507-8452
| | - Louito Edje
- L. Edje is professor of family and community medicine, Department of Medical Education and Family and Community Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio
| | - Eric J Warm
- E.J. Warm is professor of internal medicine and program director, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0002-6088-2434
| | - Hannah L Anderson
- H.L. Anderson is clinical research associate, Department of Pediatrics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0002-9435-1535
| |
Collapse
|
3
|
Kinnear B, Santen SA, Kelleher M, Martini A, Ferris S, Edje L, Warm EJ, Schumacher DJ. How Does TIMELESS Training Impact Resident Motivation for Learning, Assessment, and Feedback? Evaluating a Competency-Based Time-Variable Training Pilot. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:828-835. [PMID: 36656286 DOI: 10.1097/acm.0000000000005147] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/17/2023]
Abstract
PURPOSE As competency-based medical education has become the predominant graduate medical education training model, interest in time-variable training has grown. Despite multiple competency-based time-variable training (CBTVT) pilots ongoing in the United States, little is known about how this training approach impacts learners. The authors aim to explore how their CBTVT pilot program impacted resident motivation for learning, assessment, and feedback. METHOD The authors performed a qualitative educational case study on the Transitioning in Internal Medicine Education Leveraging Entrustment Scores Synthesis (TIMELESS) program at the University of Cincinnati from October 2020 through March 2022. Semistructured interviews were conducted with TIMELESS residents (n = 9) approximately every 6 months to capture experiences over time. The authors used inductive thematic analysis to develop themes and compared their findings with existing theories of learner motivation. RESULTS The authors developed 2 themes: TIMELESS had variable effects on residents' motivation for learning and TIMELESS increased resident engagement with and awareness of the program of assessment. Participants reported increased motivation to learn and seek assessment, though some felt a tension between performance (e.g., advancement through the residency program) and growth (e.g., improvement as a physician). Participants became more aware of the quality of assessments they received, in part due to TIMELESS increasing the perceived stakes of assessment, and reported being more deliberate when assessing other residents. CONCLUSIONS Resident motivation for learning, assessment, and feedback was impacted in ways that the authors contextualize using current theories of learner motivation (i.e., goal orientation theory and attribution theory). Future research should investigate how interventions, such as coaching, guided learner reflection, or various CBTVT implementation strategies, can help keep learners oriented toward mastery learning rather than toward performance.
Collapse
Affiliation(s)
- Benjamin Kinnear
- B. Kinnear is associate professor of internal medicine and pediatrics, Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0003-0052-4130
| | - Sally A Santen
- S.A. Santen is professor of emergency medicine, Department of Emergency Medicine, Virginia Commonwealth University School of Medicine, Richmond, Virginia, and University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0002-8327-8002
| | - Matthew Kelleher
- M. Kelleher is assistant professor of internal medicine and pediatrics, Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0002-6400-1745
| | - Abigail Martini
- A. Martini is a clinical research coordinator with emergency medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio
| | - Sarah Ferris
- S. Ferris is a research administrator, Clinical Trials Unit, Michigan Medicine Research, University of Michigan, Ann Arbor, Michigan
| | - Louito Edje
- L. Edje is professor of family and community medicine, Departments of Medical Education and of Family and Community Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio
| | - Eric J Warm
- E.J. Warm is professor of internal medicine and program director, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0002-6088-2434
| | - Daniel J Schumacher
- D.J. Schumacher is professor of pediatrics, Cincinnati Children's Hospital Medical Center and Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0001-5507-8452
| |
Collapse
|
4
|
Tay AZ, Tang PY, New LM, Zhang X, Leow WQ. Detecting residents at risk of attrition - A Singapore pathology residency's experience. Acad Pathol 2023; 10:100075. [PMID: 37095782 PMCID: PMC10121803 DOI: 10.1016/j.acpath.2023.100075] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2022] [Revised: 01/21/2023] [Accepted: 02/05/2023] [Indexed: 04/26/2023] Open
Abstract
The SingHealth Pathology Residency Program (SHPRP) is a 5-year postgraduate training program in Singapore. We face the problem of resident attrition, which has a significant impact on the individual, program and healthcare providers. Our residents are regularly evaluated, using in-house evaluations as well as assessments required in our partnership with the Accreditation Council for Graduate Medical Education International (ACGME-I). We hence sought to determine if these assessments were able to distinguish residents who would attrite from residents who would graduate successfully. Retrospective analysis of existing residency assessments was performed on all residents who have separated from SHPRP and compared with residents currently in senior residency or graduated from the program. Statistical analysis was performed on quantitative assessment methods of Resident In-Service Examination (RISE), 360-degree feedback, faculty assessment, Milestones and our own annual departmental mock examination. Word frequency analysis of narrative feedback from faculty assessment was used to generate themes. Since 2011, 10 out of 34 residents have separated from the program. RISE, Milestone data and the departmental mock examination showed statistical significance in discriminating residents at risk of attrition for specialty-related reasons from successful residents. Analysis of narrative feedback showed that successful residents performed better in areas of organization, preparation with clinical history, application of knowledge, interpersonal communication and achieving sustained progress. Existing assessment methods used in our pathology residency program are effective in detecting residents at risk of attrition. This also suggests applications in the way that we select, assess and teach residents.
Collapse
Affiliation(s)
- Amos Z.E. Tay
- Department of Anatomic Pathology, Singapore General Hospital, Singapore
- Duke-NUS Medical School, Singapore
- Corresponding author. Department of Anatomic Pathology, Singapore General Hospital, Academia, Level 10, Diagnostic Tower, 20 College Road, Singapore, 169856, Singapore.
| | - Po Yin Tang
- Department of Anatomic Pathology, Singapore General Hospital, Singapore
- Duke-NUS Medical School, Singapore
| | - Lee May New
- Department of Anatomic Pathology, Singapore General Hospital, Singapore
| | - Xiaozhu Zhang
- Department of Anatomic Pathology, Singapore General Hospital, Singapore
| | - Wei-Qiang Leow
- Department of Anatomic Pathology, Singapore General Hospital, Singapore
- Duke-NUS Medical School, Singapore
| |
Collapse
|
5
|
Early Outcomes from a Pediatric Education Research Unit. J Pediatr 2022; 249:3-5.e1. [PMID: 35227756 DOI: 10.1016/j.jpeds.2022.02.044] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/22/2022] [Accepted: 02/22/2022] [Indexed: 11/21/2022]
|
6
|
Sullivan GM, Simpson D, Artino AR, Yarris LM, Deiorio NM. The Best Graduate Medical Education Articles From 2021-in Our (Humble) Opinions. J Grad Med Educ 2022; 14:4-9. [PMID: 35222811 PMCID: PMC8848872 DOI: 10.4300/jgme-d-21-01209.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/03/2023] Open
Affiliation(s)
- Gail M. Sullivan
- Gail M. Sullivan, MD, MPH, is Editor-in-Chief, Journal of Graduate Medical Education (JGME), and Associate Director for Education, Center on Aging, and Professor of Medicine, University of Connecticut Health Center
| | - Deborah Simpson
- Deborah Simpson, PhD, is Director of Education, Academic Affairs at Advocate Aurora Health, and Clinical Adjunct Professor of Family & Community Medicine, Medical College of Wisconsin, University of Wisconsin School of Medicine and Public Health, and Deputy Editor, JGME
| | - Anthony R. Artino
- Anthony R. Artino Jr, PhD, is Professor and Associate Dean for Evaluation and Educational Research, The George Washington University School of Medicine and Health Sciences, and Deputy Editor, JGME
| | - Lalena M. Yarris
- Lalena M. Yarris, MD, MCR, is Professor of Emergency Medicine, Oregon Health & Science University, and Deputy Editor, JGME
| | - Nicole M. Deiorio
- Nicole M. Deiorio, MD, is Professor, Department of Emergency Medicine, and Associate Dean, Student Affairs, Virginia Commonwealth University School of Medicine, and Executive Editor, JGME
| |
Collapse
|