1
|
Cullen MJ, Zhang C, Sackett PR, Thakker K, Young JQ. Can a Situational Judgment Test Identify Trainees at Risk of Professionalism Issues? A Multi-Institutional, Prospective Cohort Study. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:1494-1503. [PMID: 35612909 DOI: 10.1097/acm.0000000000004756] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
PURPOSE To determine whether overall situational judgment test (SJT) scores are associated with programs' clinical competency committee (CCC) ratings of trainee professionalism, any concerning behavior, and concerning behavior requiring active remediation at 2 time periods. METHOD In fall 2019, trainees from 17 U.S. programs (16 residency, 1 fellowship) took an online 15-scenario SJT developed to measure 7 dimensions of professionalism. CCC midyear and year-end (6 months and 1 year following SJT completion, respectively) professionalism scores and concern ratings were gathered for academic year 2019-2020. Analyses were conducted to determine whether overall SJT scores related to overall professionalism ratings, trainees displaying any concerns, and trainees requiring active remediation at both time periods. RESULTS Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings ( r = .21 and .14, P < .001 and = .03, respectively). Holding gender and race/ethnicity constant, a 1 standard deviation (SD) increase in overall SJT score was associated with a .20 SD increase in overall professionalism ratings at midyear ( P = .005) and a .22 SD increase at year-end ( P = .001). Holding gender and race/ethnicity constant, a 1 SD increase in overall SJT score decreased the odds of a trainee displaying any concerns by 37% (odds ratio [OR] 95% confidence interval [CI]: [.44, .87], P = .006) at midyear and 34% (OR 95% CI: [.46, .95], P = .025) at year-end and decreased the odds of a trainee requiring active remediation by 51% (OR 95% CI: [.25, .90], P = .02) at midyear. CONCLUSIONS Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings and were associated with whether trainees exhibited any concerning behavior at midyear and year-end and whether trainees needed active remediation at midyear. Future research should investigate whether other potential professionalism measures are associated with concerning trainee behavior.
Collapse
Affiliation(s)
- Michael J Cullen
- M.J. Cullen is senior director of assessment, evaluation, and research for graduate medical education, University of Minnesota Medical School, Minneapolis, Minnesota; ORCID: https://orcid.org/0000-0002-4755-4276
| | - Charlene Zhang
- C. Zhang was a graduate student, Industrial/Organizational Psychology Program, University of Minnesota-Twin Cities, Minneapolis, Minnesota, at the time of the study. The author is now a research scientist, Amazon, Alexandria, Virginia; ORCID: http://orcid.org/0000-0001-6975-5653
| | - Paul R Sackett
- P.R. Sackett is professor of psychology, Industrial/Organizational Psychology Program, University of Minnesota-Twin Cities, Minneapolis, Minnesota; ORCID: http://orcid.org/0000-0001-7633-4160
| | - Krima Thakker
- K. Thakker is research coordinator, Zucker Hillside Hospital, Northwell Health, Glen Oaks, New York; ORCID: https://orcid.org/0000-0002-1737-2113
| | - John Q Young
- J.Q. Young is professor and chair, Department of Psychiatry, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York, and senior vice president, behavioral health, Northwell Health, Lake Success, New York; ORCID: https://orcid.org/0000-0003-2219-5657
| |
Collapse
|
2
|
Cheong CWS, Quah ELY, Chua KZY, Lim WQ, Toh RQE, Chiang CLL, Ng CWH, Lim EG, Teo YH, Kow CS, Vijayprasanth R, Liang ZJ, Tan YKI, Tan JRM, Chiam M, Lee ASI, Ong YT, Chin AMC, Wijaya L, Fong W, Mason S, Krishna LKR. Post graduate remediation programs in medicine: a scoping review. BMC MEDICAL EDUCATION 2022; 22:294. [PMID: 35443679 PMCID: PMC9020048 DOI: 10.1186/s12909-022-03278-x] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/26/2021] [Accepted: 03/16/2022] [Indexed: 06/14/2023]
Abstract
BACKGROUND Recognizing that physicians may struggle to achieve knowledge, skills, attitudes and or conduct at one or more stages during their training has highlighted the importance of the 'deliberate practice of improving performance through practising beyond one's comfort level under guidance'. However, variations in physician, program, contextual and healthcare and educational systems complicate efforts to create a consistent approach to remediation. Balancing the inevitable disparities in approaches and settings with the need for continuity and effective oversight of the remediation process, as well as the context and population specific nature of remediation, this review will scrutinise the remediation of physicians in training to better guide the design, structuring and oversight of new remediation programs. METHODS Krishna's Systematic Evidence Based Approach is adopted to guide this Systematic Scoping Review (SSR in SEBA) to enhance the transparency and reproducibility of this review. A structured search for articles on remediation programs for licenced physicians who have completed their pre-registration postings and who are in training positions published between 1st January 1990 and 31st December 2021 in PubMed, Scopus, ERIC, Google Scholar, PsycINFO, ASSIA, HMIC, DARE and Web of Science databases was carried out. The included articles were concurrently thematically and content analysed using SEBA's Split Approach. Similarities in the identified themes and categories were combined in the Jigsaw Perspective and compared with the tabulated summaries of included articles in the Funnelling Process to create the domains that will guide discussions. RESULTS The research team retrieved 5512 abstracts, reviewed 304 full-text articles and included 101 articles. The domains identified were characteristics, indications, frameworks, domains, enablers and barriers and unique features of remediation in licenced physicians in training programs. CONCLUSION Building upon our findings and guided by Hauer et al. approach to remediation and Taylor and Hamdy's Multi-theories Model, we proffer a theoretically grounded 7-stage evidence-based remediation framework to enhance understanding of remediation in licenced physicians in training programs. We believe this framework can guide program design and reframe remediation's role as an integral part of training programs and a source of support and professional, academic, research, interprofessional and personal development.
Collapse
Affiliation(s)
- Clarissa Wei Shuen Cheong
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Elaine Li Ying Quah
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Keith Zi Yuan Chua
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Wei Qiang Lim
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Rachelle Qi En Toh
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Christine Li Ling Chiang
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Caleb Wei Hao Ng
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Elijah Gin Lim
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Yao Hao Teo
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Cheryl Shumin Kow
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Raveendran Vijayprasanth
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Zhen Jonathan Liang
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Yih Kiat Isac Tan
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Javier Rui Ming Tan
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Min Chiam
- Division of Cancer Education, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 169610 Singapore
| | - Alexia Sze Inn Lee
- Division of Cancer Education, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 169610 Singapore
| | - Yun Ting Ong
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
| | - Annelissa Mien Chew Chin
- Medical Library, National University of Singapore Libraries, Blk MD6, Centre, 14 Medical Dr, #05-01 for Translational Medicine, Singapore, 117599 Singapore
| | - Limin Wijaya
- Duke-NUS Medical School, 8 College Road, Singapore, 169857 Singapore
- Department of Infectious Diseases, Singapore General Hospital, Outram Road, Singapore, 169608 Singapore
| | - Warren Fong
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Duke-NUS Medical School, 8 College Road, Singapore, 169857 Singapore
- Department of Rheumatology and Immunology, Singapore General Hospital, 16 College Road, Block 6 Level 9, Singapore, 169854 Singapore
| | - Stephen Mason
- Palliative Care Institute Liverpool, Academic Palliative & End of Life Care Centre, Cancer Research Centre, University of Liverpool, 200 London Road, Liverpool, L3 9TA UK
| | - Lalit Kumar Radha Krishna
- Yong Loo Lin School of Medicine, National University Singapore, 1E Kent Ridge Road, 119228 NUHS Tower Block, Level, Singapore, 11 Singapore
- Division of Supportive Palliative and Care, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 16961 Singapore
- Division of Cancer Education, National Cancer Centre Singapore, 11 Hospital Crescent, Singapore, 169610 Singapore
- Duke-NUS Medical School, 8 College Road, Singapore, 169857 Singapore
- Palliative Care Institute Liverpool, Academic Palliative & End of Life Care Centre, Cancer Research Centre, University of Liverpool, 200 London Road, Liverpool, L3 9TA UK
- Centre for Biomedical Ethics, National University of Singapore, Blk MD11, 10 Medical Drive, #02-03, Singapore, 117597 Singapore
- PalC, The Palliative Care Centre for Excellence in Research and Education, PalC c/o Dover Park Hospice, 10 Jalan Tan Tock Seng, Singapore, 308436 Singapore
| |
Collapse
|
3
|
Price T, Wong G, Withers L, Wanner A, Cleland J, Gale T, Prescott-Clements L, Archer J, Bryce M, Brennan N. Optimising the delivery of remediation programmes for doctors: A realist review. MEDICAL EDUCATION 2021; 55:995-1010. [PMID: 33772829 DOI: 10.1111/medu.14528] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/24/2020] [Revised: 03/16/2021] [Accepted: 03/19/2021] [Indexed: 05/15/2023]
Abstract
CONTEXT Medical underperformance puts patient safety at risk. Remediation, the process that seeks to 'remedy' underperformance and return a doctor to safe practice, is therefore a crucially important area of medical education. However, although remediation is used in health care systems globally, there is limited evidence for the particular models or strategies employed. The purpose of this study was to conduct a realist review to ascertain why, how, in what contexts, for whom and to what extent remediation programmes for practising doctors work to restore patient safety. METHOD We conducted a realist literature review consistent with RAMESES standards. We developed a programme theory of remediation by carrying out a systematic search of the literature and through regular engagement with a stakeholder group. We searched bibliographic databases (MEDLINE, EMBASE, PsycINFO, HMIC, CINAHL, ERIC, ASSIA and DARE) and conducted purposive supplementary searches. Relevant sections of text relating to the programme theory were extracted and synthesised using a realist logic of analysis to identify context-mechanism-outcome configurations (CMOcs). RESULTS A 141 records were included. The majority of the studies were from North America (64%). 29 CMOcs were identified. Remediation programmes are effective when a doctor's insight and motivation are developed and behaviour change reinforced. Insight can be developed by providing safe spaces, using advocacy to promote trust and framing feedback sensitively. Motivation can be enhanced by involving the doctor in remediation planning, correcting causal attribution, goal setting and destigmatising remediation. Sustained change can be achieved by practising new behaviours and skills, and through guided reflection. CONCLUSION Remediation can work when it creates environments that trigger behaviour change mechanisms. Our evidence synthesis provides detailed recommendations on tailoring implementation and design strategies to improve remediation interventions for doctors.
Collapse
Affiliation(s)
- Tristan Price
- Collaboration for the Advancement of Medical Education Research and Assessment, Faculty of Health, University of Plymouth, Plymouth, UK
| | - Geoff Wong
- Nuffield Department of Primary Care, Health Sciences, University of Oxford, Oxford, UK
| | | | - Amanda Wanner
- NIHR Collaboration for Leadership in Applied Health Research and Care South West Peninsula (PenCLAHRC), Community and Primary Care Research Group, University of Plymouth, Plymouth, UK
| | - Jennifer Cleland
- Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore City, Singapore
| | - Tom Gale
- Collaboration for the Advancement of Medical Education Research and Assessment, Faculty of Health, University of Plymouth, Plymouth, UK
| | | | - Julian Archer
- Faculty of Medicine, Nursing and Healthcare, Monash University, Melbourne, Vic., Australia
| | - Marie Bryce
- Collaboration for the Advancement of Medical Education Research and Assessment, Faculty of Health, University of Plymouth, Plymouth, UK
| | - Nicola Brennan
- Collaboration for the Advancement of Medical Education Research and Assessment, Faculty of Health, University of Plymouth, Plymouth, UK
| |
Collapse
|
4
|
Price T, Brennan N, Wong G, Withers L, Cleland J, Wanner A, Gale T, Prescott-Clements L, Archer J, Bryce M. Remediation programmes for practising doctors to restore patient safety: the RESTORE realist review. HEALTH SERVICES AND DELIVERY RESEARCH 2021. [DOI: 10.3310/hsdr09110] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022] Open
Abstract
Background
An underperforming doctor puts patient safety at risk. Remediation is an intervention intended to address underperformance and return a doctor to safe practice. Used in health-care systems all over the world, it has clear implications for both patient safety and doctor retention in the workforce. However, there is limited evidence underpinning remediation programmes, particularly a lack of knowledge as to why and how a remedial intervention may work to change a doctor’s practice.
Objectives
To (1) conduct a realist review of the literature to ascertain why, how, in what contexts, for whom and to what extent remediation programmes for practising doctors work to restore patient safety; and (2) provide recommendations on tailoring, implementation and design strategies to improve remediation interventions for doctors.
Design
A realist review of the literature underpinned by the Realist And MEta-narrative Evidence Syntheses: Evolving Standards quality and reporting standards.
Data sources
Searches of bibliographic databases were conducted in June 2018 using the following databases: EMBASE, MEDLINE, Cumulative Index to Nursing and Allied Health Literature, PsycINFO, Education Resources Information Center, Database of Abstracts of Reviews of Effects, Applied Social Sciences Index and Abstracts, and Health Management Information Consortium. Grey literature searches were conducted in June 2019 using the following: Google Scholar (Google Inc., Mountain View, CA, USA), OpenGrey, NHS England, North Grey Literature Collection, National Institute for Health and Care Excellence Evidence, Electronic Theses Online Service, Health Systems Evidence and Turning Research into Practice. Further relevant studies were identified via backward citation searching, searching the libraries of the core research team and through a stakeholder group.
Review methods
Realist review is a theory-orientated and explanatory approach to the synthesis of evidence that seeks to develop programme theories about how an intervention produces its effects. We developed a programme theory of remediation by convening a stakeholder group and undertaking a systematic search of the literature. We included all studies in the English language on the remediation of practising doctors, all study designs, all health-care settings and all outcome measures. We extracted relevant sections of text relating to the programme theory. Extracted data were then synthesised using a realist logic of analysis to identify context–mechanism–outcome configurations.
Results
A total of 141 records were included. Of the 141 studies included in the review, 64% related to North America and 14% were from the UK. The majority of studies (72%) were published between 2008 and 2018. A total of 33% of articles were commentaries, 30% were research papers, 25% were case studies and 12% were other types of articles. Among the research papers, 64% were quantitative, 19% were literature reviews, 14% were qualitative and 3% were mixed methods. A total of 40% of the articles were about junior doctors/residents, 31% were about practicing physicians, 17% were about a mixture of both (with some including medical students) and 12% were not applicable. A total of 40% of studies focused on remediating all areas of clinical practice, including medical knowledge, clinical skills and professionalism. A total of 27% of studies focused on professionalism only, 19% focused on knowledge and/or clinical skills and 14% did not specify. A total of 32% of studies described a remediation intervention, 16% outlined strategies for designing remediation programmes, 11% outlined remediation models and 41% were not applicable. Twenty-nine context–mechanism–outcome configurations were identified. Remediation programmes work when they develop doctors’ insight and motivation, and reinforce behaviour change. Strategies such as providing safe spaces, using advocacy to develop trust in the remediation process and carefully framing feedback create contexts in which psychological safety and professional dissonance lead to the development of insight. Involving the remediating doctor in remediation planning can provide a perceived sense of control in the process and this, alongside correcting causal attribution, goal-setting, destigmatising remediation and clarity of consequences, helps motivate doctors to change. Sustained change may be facilitated by practising new behaviours and skills and through guided reflection.
Limitations
Limitations were the low quality of included literature and limited number of UK-based studies.
Future work
Future work should use the recommendations to optimise the delivery of existing remediation programmes for doctors in the NHS.
Study registration
This study is registered as PROSPERO CRD42018088779.
Funding
This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 9, No. 11. See the NIHR Journals Library website for further project information.
Collapse
Affiliation(s)
- Tristan Price
- Collaboration for the Advancement of Medical Education Research and Assessment (CAMERA), Faculty of Health, University of Plymouth, Plymouth, UK
| | - Nicola Brennan
- Collaboration for the Advancement of Medical Education Research and Assessment (CAMERA), Faculty of Health, University of Plymouth, Plymouth, UK
| | - Geoff Wong
- Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
| | | | - Jennifer Cleland
- Medical Education Research and Scholarship Unit (MERSU), Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore
| | - Amanda Wanner
- Collaboration for the Advancement of Medical Education Research and Assessment (CAMERA), Faculty of Health, University of Plymouth, Plymouth, UK
| | - Thomas Gale
- Collaboration for the Advancement of Medical Education Research and Assessment (CAMERA), Faculty of Health, University of Plymouth, Plymouth, UK
| | | | - Julian Archer
- Medicine, Nursing and Health Sciences Education Portfolio, Monash University, Melbourne, VIC, Australia
| | - Marie Bryce
- Collaboration for the Advancement of Medical Education Research and Assessment (CAMERA), Faculty of Health, University of Plymouth, Plymouth, UK
| |
Collapse
|
5
|
Rayamajhi S, Dhakal P, Wang L, Rai MP, Shrotriya S. Do USMLE steps, and ITE score predict the American Board of Internal Medicine Certifying Exam results? BMC MEDICAL EDUCATION 2020; 20:79. [PMID: 32183789 PMCID: PMC7079442 DOI: 10.1186/s12909-020-1974-3] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/12/2019] [Accepted: 02/20/2020] [Indexed: 05/25/2023]
Abstract
BACKGROUND To evaluate if United States Medical Licensing Examination (USMLE) Step 1, USMLE Step 2 CK, USMLE Step 3, and residency third-year in-service training exam (ITE) scores predict the results of American Board of Internal Medicine Certifying Exam (ABIM-CE). METHODS We performed a retrospective review of USMLE Step 1, USMLE Step 2 CK, USMLE Step 3, third-year residency ITE scores and ABIM-CE results of IM residents at our residency program from 2004 through 2017. Statistical analysis was perfrormed using Pearson correlation coefficient, and logistic regression to assess the relationship between USMLE Step 1, USMLE Step 2CK, USMLE Step 3, 3rd year ITE scores and ABIM-CE results. We used Multivariate logistic regression to predict pass or fail results in ABIM-CE based on USMLE and third-year ITE test scores controlling for other covariates. RESULTS Among 114 Internal Medicine MD residents included in the study, 92% (n = 105) passed the ABIM-CE. The OR of passing ABIM-CE was 2.70 (95% CI = 1.38-5.29), 2.31 (95% CI = 1.33-4.01), and 1.63 (95% CI = 0.81-3.29) with a ten-point increase in USMLE Step 1, USMLE Step 2 CK and USMLE Step 3 scores respectively. The OR of ABIM-CE passing chance was 2.96 (95% CI = 0.95-9.20), with a ten-point increase in the average score of the above three exams. A 5 % increase in ITE percentage raised the likelihood of passing ABIM-CE (OR 2.92, 95% CI 1.15-7.38). All residents who failed ABIM-CE had Step 1 scores < 220. Among 31 residents with Step 2 CK score < 220, 20% (n = 6) failed ABIM. Similarly, 9% of residents with USMLE Step 3 score < 220 failed ABIM-CE; all residents who failed had scored < 220. The probability curve predicted that the chance of passing ABIM- CE was around 80% with USMLE scores greater than 200 and increased to almost 100% with USMLE scores of 250 or more. CONCLUSION USMLE Step 1, USMLE Step 2 CK, and third-year ITE scores can predict the chances of passing ABIM-CE. The third-year ITE score has a higher preditive value compared to USMLE Step 1 and USMLE Step 2 scores. USMLE Step 1 scores more predictive of ABIM-CE results compared to USMLE Step 2CK scores. Thus, residency programs can identify internal medicine residents at risk of failing ABIM-CE and formulate interventions at an early stage during residency training. Measures such as enrolling them in question banks or board review courses can be helpful in improving their chances of passing ABIM-CE.
Collapse
Affiliation(s)
- Supratik Rayamajhi
- Department of Medicine, Michigan State University, 788 Service Road, Room B301 Clinical Center, East Lansing, MI, 48824, USA
| | - Prajwal Dhakal
- Division of Oncology and Hematology, Department of Internal Medicine, University of Nebraska Medical Center, Omaha, NE, USA
- Fred and Pamela Buffett Cancer Center, University of Nebraska Medical Center, Omaha, NE, USA
| | - Ling Wang
- Department of Medicine, Michigan State University, 788 Service Road, Room B301 Clinical Center, East Lansing, MI, 48824, USA
| | - Manoj P Rai
- Department of Medicine, Michigan State University, 788 Service Road, Room B301 Clinical Center, East Lansing, MI, 48824, USA.
| | - Shiva Shrotriya
- Department of Medicine, Michigan State University, 788 Service Road, Room B301 Clinical Center, East Lansing, MI, 48824, USA
| |
Collapse
|
6
|
Hauer KE, Vandergrift J, Lipner RS, Holmboe ES, Hood S, McDonald FS. National Internal Medicine Milestone Ratings: Validity Evidence From Longitudinal Three-Year Follow-up. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2018; 93:1189-1204. [PMID: 29620673 DOI: 10.1097/acm.0000000000002234] [Citation(s) in RCA: 21] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
PURPOSE To evaluate validity evidence for internal medicine milestone ratings across programs for three resident cohorts by quantifying "not assessable" ratings; reporting mean longitudinal milestone ratings for individual residents; and correlating medical knowledge ratings across training years with certification examination scores to determine predictive validity of milestone ratings for certification outcomes. METHOD This retrospective study examined milestone ratings for postgraduate year (PGY) 1-3 residents in U.S. internal medicine residency programs. Data sources included milestone ratings, program characteristics, and certification examination scores. RESULTS Among 35,217 participants, there was a decreased percentage with "not assessable" ratings across years: 1,566 (22.5%) PGY1s in 2013-2014 versus 1,219 (16.6%) in 2015-2016 (P = .01), and 342 (5.1%) PGY3s in 2013-2014 versus 177 (2.6%) in 2015-2016 (P = .04). For individual residents with three years of ratings, mean milestone ratings increased from around 3 (behaviors of an early learner or advancing resident) in PGY1 (ranging from a mean of 2.73 to 3.19 across subcompetencies) to around 4 (ready for unsupervised practice) in PGY3 (mean of 4.00 to 4.22 across subcompetencies, P < .001 for all subcompetencies). For each increase of 0.5 units in two medical knowledge (MK1, MK2) subcompetency ratings, the difference in examination scores for PGY3s was 19.5 points for MK1 (P < .001) and 19.0 for MK2 (P < .001). CONCLUSIONS These findings provide evidence of validity of the milestones by showing how training programs have applied them over time and how milestones predict other training outcomes.
Collapse
Affiliation(s)
- Karen E Hauer
- K.E. Hauer is associate dean for assessment and professor, Department of Medicine, University of California at San Francisco, San Francisco, California. J. Vandergrift is a health services researcher, American Board of Internal Medicine (ABIM), Philadelphia, Pennsylvania. R.S. Lipner is senior vice president of assessment and research, ABIM, Philadelphia, Pennsylvania. E.S. Holmboe is senior vice president of milestones development and evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois. S. Hood is director of initial certification, ABIM, Philadelphia, Pennsylvania. F.S. McDonald is senior vice president of academic and medical affairs, ABIM, Philadelphia, Pennsylvania
| | | | | | | | | | | |
Collapse
|
7
|
Shah KH, Jordan J, Jahnes K, Lisbon DP, Lutfy-Clayton L, Wei G, Winkel G, Santen SA. Audience Response System Facilitates Prediction of Scores on In-Training Examination. West J Emerg Med 2017; 18:525-530. [PMID: 28435506 PMCID: PMC5391905 DOI: 10.5811/westjem.2017.1.32858] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2016] [Revised: 12/15/2016] [Accepted: 01/09/2017] [Indexed: 11/11/2022] Open
Abstract
Introduction Audience response systems (ARS) are increasingly popular; however, their contribution to education is not completely clear. Our study found that scores from review quizzes delivered by an ARS correlate with in-training exam (ITE) scores and are viewed positively by residents. This information may be useful in identifying poor performers early so that targeted educational interventions can be made. The objective was to determine if scores on review quizzes delivered by an ARS correlate with ITE scores and to obtain participant feedback on use of the ARS for ITE preparation. Methods This was a prospective observational study of emergency medicine (EM) residents at six accredited EM residency programs. Subjects included residents who had taken previous ITEs. Subjects participated in bimonthly review sessions using an ARS. Twelve review quizzes were administered, each consisting of 10 multiple-choice questions. After the ITE, subjects completed an attitudinal survey consisting of six Likert-scale items and one “yes/no” item. We used a mixed linear model to analyze the data, accounting for prior 2012 ITE scores and nesting due to institution. Results Among 192 participants, 135 (70.3%) completed the ITE in both 2012 and 2013; we analyzed their data for the first objective. Results from the mixed linear model indicate that the total mean score on the review quizzes was a significant [t(127) = 6.68; p < 0.001] predictor of the 2013 ITE after controlling for the 2012 ITE score. One hundred forty-six (76.0%) participants completed the attitudinal survey; 96% of respondents stated that they would like ARS to be used more often in resident education. Respondents felt the sessions aided in learning (mean 7.7/10), assisted in preparation for the ITE (mean 6.7/10), and helped identify content areas of weakness (mean 7.6/10). Conclusion Our results suggest that scores from review quizzes delivered by an audience response system correlate with in-training exam scores and is viewed positively by residents.
Collapse
Affiliation(s)
- Kaushal H Shah
- Icahn School of Medicine at Mt. Sinai, Department of Emergency Medicine, New York, New York
| | - Jaime Jordan
- Harbor-UCLA Medical Center, Department of Emergency Medicine, Torrance, California
| | - Katherine Jahnes
- NYU Langone Medical Center, Ronald O. Perelman Department of Emergency Medicine, New York, New York
| | - David P Lisbon
- University of Kansas Hospital, Department of Emergency Medicine, Kansas City, Kansas
| | - Lucienne Lutfy-Clayton
- University of Massachusetts Medical School - Baystate Health, Department of Emergency Medicine, Springfield, Massachusetts
| | - Grant Wei
- Robert Wood Johnson University Hospital, Department of Emergency Medicine, New Brunswick, New Jersey
| | - Gary Winkel
- Icahn School of Medicine at Mt. Sinai, Department of Emergency Medicine, New York, New York
| | - Sally A Santen
- University of Michigan, Department of Emergency Medicine, Ann Arbor, Michigan
| |
Collapse
|
8
|
Smith JL, Lypson M, Silverberg M, Weizberg M, Murano T, Lukela M, Santen SA. Defining Uniform Processes for Remediation, Probation and Termination in Residency Training. West J Emerg Med 2016; 18:110-113. [PMID: 28116019 PMCID: PMC5226740 DOI: 10.5811/westjem.2016.10.31483] [Citation(s) in RCA: 18] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/07/2016] [Revised: 10/03/2016] [Accepted: 10/26/2016] [Indexed: 11/11/2022] Open
Abstract
It is important that residency programs identify trainees who progress appropriately, as well as identify residents who fail to achieve educational milestones as expected so they may be remediated. The process of remediation varies greatly across training programs, due in part to the lack of standardized definitions for good standing, remediation, probation, and termination. The purpose of this educational advancement is to propose a clear remediation framework including definitions, management processes, documentation expectations and appropriate notifications. Informal remediation is initiated when a resident's performance is deficient in one or more of the outcomes-based milestones established by the Accreditation Council for Graduate Medical Education, but not significant enough to trigger formal remediation. Formal remediation occurs when deficiencies are significant enough to warrant formal documentation because informal remediation failed or because issues are substantial. The process includes documentation in the resident's file and notification of the graduate medical education office; however, the documentation is not disclosed if the resident successfully remediates. Probation is initiated when a resident is unsuccessful in meeting the terms of formal remediation or if initial problems are significant enough to warrant immediate probation. The process is similar to formal remediation but also includes documentation extending to the final verification of training and employment letters. Termination involves other stakeholders and occurs when a resident is unsuccessful in meeting the terms of probation or if initial problems are significant enough to warrant immediate termination.
Collapse
Affiliation(s)
- Jessica L Smith
- Alpert Medical School of Brown University, Department of Emergency Medicine, Providence, Rhode Island
| | - Monica Lypson
- University of Michigan, Department of Internal Medicine, Ann Arbor, Michigan
| | - Mark Silverberg
- SUNY Downstate/Kings County Hospital, Department of Emergency Medicine, Brooklyn, New York
| | - Moshe Weizberg
- Staten Island University Hospital, Northwell Health, Staten Island, New York
| | - Tiffany Murano
- Columbia University Medical College-NY Presbyterian Hospital, Department of Emergency Medicine, New York, New York
| | - Michael Lukela
- University of Michigan, Department of Internal Medicine, Ann Arbor, Michigan; University of Michigan, Department of Pediatrics, Ann Arbor, Michigan
| | - Sally A Santen
- University of Michigan, Department of Learning Health Sciences, Ann Arbor, Michigan
| |
Collapse
|
9
|
Marco CA, Wahl RP, Counselman FL, Heller BN, Kowalenko T, Harvey AL, Joldersma KB, Reisdorff EJ. Physician Preparation for the American Board of Emergency Medicine ConCert Examination. Acad Emerg Med 2016; 23:191-6. [PMID: 26802600 DOI: 10.1111/acem.12866] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2015] [Revised: 08/14/2015] [Accepted: 08/19/2015] [Indexed: 11/28/2022]
Abstract
OBJECTIVES To maintain certification by the American Board of Emergency Medicine (ABEM), physicians are required to pass the Continuous Certification (ConCert) examination at least every 10 years. On the 2014 ConCert postexamination survey, ABEM sought to understand the manner in which ABEM diplomates prepared for the test and to identify associations between test preparation approaches and performance on the ConCert examination. METHODS This was a cross-sectional survey study. The survey was administered at the end of the 2014 ConCert examination. Analyses included chi-square and linear regression to determine the association of preparation methods with performance. RESULTS Of the 2,431 on-time test-takers, 2,338 (96.2%) were included. The most commonly used study approach was the review of written materials designed for test preparation (1,585; 67.8%), followed by an online training course (1,006; 43.0%). There were 758 (32.4%) physicians who took a single onsite board review course, while 41 (1.8%) took two or more onsite courses. Most physicians (1,611; 68.9%) spent over 35 hours preparing for the ConCert examination. The study method that was most associated with favorable test scores was the review of written materials designed for test preparation (p < 0.001). Attending an onsite preparation course was associated with poorer performance (p < 0.001). There was a significant association between no additional preparation and failing the examination (chi-square with Yates correction; p = 0.001). CONCLUSIONS A substantial majority (97.8%) of physicians taking the 2014 ABEM ConCert examination prepared for it. The majority of physicians used written materials specifically designed for test preparation. Reviewing written materials designed for test preparation was associated with the highest performance.
Collapse
Affiliation(s)
- Catherine A. Marco
- Department of Emergency Medicine; Wright State University Boonshoft School of Medicine; Dayton OH
| | - Robert P. Wahl
- Department of Emergency Medicine; Wayne State University School of Medicine; Detroit MI
| | - Francis L. Counselman
- Department of Emergency Medicine; Eastern Virginia Medical School and Emergency Physicians of Tidewater; Norfolk VA
| | - Barry N. Heller
- Department of Emergency Medicine; St. Mary Medical Center; Long Beach CA
| | - Terry Kowalenko
- Department of Emergency Medicine; Beaumont Health System; Royal Oak MI
| | | | | | | |
Collapse
|
10
|
Silverberg M, Weizberg M, Murano T, Smith JL, Burkhardt JC, Santen SA. What is the Prevalence and Success of Remediation of Emergency Medicine Residents? West J Emerg Med 2015; 16:839-44. [PMID: 26594275 PMCID: PMC4651579 DOI: 10.5811/westjem.2015.9.27357] [Citation(s) in RCA: 24] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2015] [Revised: 09/01/2015] [Accepted: 09/09/2015] [Indexed: 11/11/2022] Open
Abstract
INTRODUCTION The primary objective of this study was to determine the prevalence of remediation, competency domains for remediation, the length, and success rates of remediation in emergency medicine (EM). METHODS We developed the survey in Surveymonkey™ with attention to content and response process validity. EM program directors responded how many residents had been placed on remediation in the last three years. Details regarding the remediation were collected including indication, length and success. We reported descriptive data and estimated a multinomial logistic regression model. RESULTS We obtained 126/158 responses (79.7%). Ninety percent of programs had at least one resident on remediation in the last three years. The prevalence of remediation was 4.4%. Indications for remediation ranged from difficulties with one core competency to all six competencies (mean 1.9). The most common were medical knowledge (MK) (63.1% of residents), patient care (46.6%) and professionalism (31.5%). Mean length of remediation was eight months (range 1-36 months). Successful remediation was 59.9% of remediated residents; 31.3% reported ongoing remediation. In 8.7%, remediation was deemed "unsuccessful." Training year at time of identification for remediation (post-graduate year [PGY] 1), longer time spent in remediation, and concerns with practice-based learning (PBLI) and professionalism were found to have statistically significant association with unsuccessful remediation. CONCLUSION Remediation in EM residencies is common, with the most common areas being MK and patient care. The majority of residents are successfully remediated. PGY level, length of time spent in remediation, and the remediation of the competencies of PBLI and professionalism were associated with unsuccessful remediation.
Collapse
Affiliation(s)
- Mark Silverberg
- SUNY Downstate/Kings County Hospital, Department of Emergency Medicine, New York, New York
| | - Moshe Weizberg
- Staten Island University Hospital, Department of Emergency Medicine, Staten Island, New York
| | - Tiffany Murano
- Rutgers New Jersey Medical School, Department of Emergency Medicine, Newark, New Jersey
| | - Jessica L Smith
- Alpert Medical School of Brown University, Department of Emergency Medicine, Providence, Rhode Island
| | - John C Burkhardt
- University of Michigan Medical School, Department of Emergency Medicine, Ann Arbor, Michigan
| | - Sally A Santen
- University of Michigan Medical School, Department of Emergency Medicine, Ann Arbor, Michigan
| |
Collapse
|
11
|
Atsawarungruangkit A. Residency program characteristics that are associated with pass rate of the American Board of Pediatrics certifying exam. ADVANCES IN MEDICAL EDUCATION AND PRACTICE 2015; 6:517-524. [PMID: 26316837 PMCID: PMC4542559 DOI: 10.2147/amep.s90022] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 06/04/2023]
Abstract
BACKGROUND The US is home to almost 200 pediatrics residency programs; despite this, there is little information about the relationship between program characteristics and performance in the American Board of Pediatrics (ABP) certifying exam. OBJECTIVE To evaluate the relationship between pass rate of the ABP certifying exam with the characteristics of categorical pediatrics residency programs. METHODS This retrospective, cross-sectional study used publicly available data from the ABP website and the Fellowship and Residency Electronic Interactive Database. All programs that reported pass rates were included. The analysis, comprising univariate and multivariate linear regression, involved determining how 69 factors (eg, general information, number of faculty and trainees, work schedule, educational environment) related to the pass rate. RESULTS Of 199 programs, 194 reported pass rates. The univariate analysis revealed 20 program characteristics with P-values <0.10. However, in the multivariate analysis, pass rate was significantly associated with only three program characteristics: ratio of full-time equivalent paid faculty to positions, percentage of US medical graduates, and average hours per week of regularly scheduled lectures or conferences. CONCLUSION Unlike in previous studies, location and program size were not significantly associated with the pass rate in this multivariate analysis. The finding regarding the ratio of full-time equivalent paid faculty to positions highlighted the benefits of a well-supervised training environment, while that regarding the percentage of US medical graduates indicated the necessity of high competition in residency programs. Finally, longer hours per week of regularly scheduled lectures or conferences were associated with better academic outcomes, both statistically and intuitively.
Collapse
|
12
|
Kay C, Jackson JL, Frank M. The relationship between internal medicine residency graduate performance on the ABIM certifying examination, yearly in-service training examinations, and the USMLE Step 1 examination. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2015; 90:100-104. [PMID: 25271892 DOI: 10.1097/acm.0000000000000500] [Citation(s) in RCA: 47] [Impact Index Per Article: 5.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/03/2023]
Abstract
PURPOSE To explore the relationship between United States Medical Licensing Examination (USMLE) Step 1 scores, yearly in-service training exam (ITE) scores, and passing the American Board of Internal Medicine certifying examination (ABIM-CE). METHOD The authors conducted a retrospective database review of internal medicine residents from the Medical College of Wisconsin from 2004 through 2012. Residents' USMLE Step 1, ITE, and ABIM-CE scores were extracted. Pearson rho, chi-square, and logistic regression were used to determine whether relationships existed between the scores and if Step 1 and ITE scores correlate with passing the ABIM-CE. RESULTS There were 241 residents, who participated in 728 annual ITEs. There were Step 1 scores for 195 (81%) residents and ABIM-CE scores for 183 (76%). Step 1 and ABIM-CE scores had a modest correlation (rho: 0.59), as did ITE and ABIM-CE scores (rho: 0.48-0.67). Failing Step 1 or being in the bottom ITE quartile during any year of testing markedly increased likelihood of failing the boards (Step 1: relative risk [RR]: 2.4; 95% CI: 1.0-5.9; first-year residents' RR: 1.3; 95% CI: 1.0-1.6; second-year residents' RR: 1.3; 95% CI: 1.1-1.5; third-year residents' RR: 1.3; 95% CI: 1.1-1.5). CONCLUSIONS USMLE Step 1 and ITE scores have a modest correlation with board scores. Failing Step 1 or scoring in the bottom quartile of the ITE increased the risk of failing the boards. What effective intervention, if any, program directors may use with at-risk residents is a question deserving further research.
Collapse
Affiliation(s)
- Cynthia Kay
- Dr. Kay is instructor/fellow, Department of Medicine, Medical College of Wisconsin, Milwaukee, Wisconsin. Dr. Jackson is professor of medicine, Medical College of Wisconsin, and chief, General Internal Medicine Section, Zablocki VA Medical Center, Milwaukee, Wisconsin. Dr. Frank is professor of medicine and program director, Internal Medicine Residency Program, Medical College of Wisconsin, Milwaukee, Wisconsin
| | | | | |
Collapse
|
13
|
Farrell SE, Kuhn GJ, Coates WC, Shayne PH, Fisher J, Maggio LA, Lin M. Critical appraisal of emergency medicine education research: the best publications of 2013. Acad Emerg Med 2014; 21:1274-83. [PMID: 25377406 DOI: 10.1111/acem.12507] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2014] [Accepted: 07/27/2014] [Indexed: 11/30/2022]
Abstract
OBJECTIVES The objective was to critically appraise and highlight methodologically superior medical education research articles published in 2013 whose outcomes are pertinent to teaching and education in emergency medicine (EM). METHODS A search of the English-language literature in 2013 querying Education Resources Information Center (ERIC), PsychINFO, PubMed, and Scopus identified 251 EM-related studies using hypothesis-testing or observational investigations of educational interventions. Two reviewers independently screened all of the publications and removed articles using established exclusion criteria. Six reviewers then independently scored the remaining 43 publications using either a qualitative a or quantitative scoring system, based on the research methodology of each article. Each scoring system consisted of nine criteria. Selected criteria were based on accepted educational review literature and chosen a priori. Both scoring systems used parallel scoring metrics and have been used previously within this annual review. RESULTS Forty-three medical education research papers (37 quantitative and six qualitative studies) met the a priori criteria for inclusion and were reviewed. Six quantitative and one qualitative study were scored and ranked most highly by the reviewers as exemplary and are summarized in this article. CONCLUSIONS This annual critical appraisal article aims to promote superior research in EM-related education, by reviewing and highlighting seven of 43 major education research studies, meeting a priori criteria, and published in 2013. Common methodologic pitfalls in the 2013 papers are noted, and current trends in medical education research in EM are discussed.
Collapse
Affiliation(s)
- Susan E. Farrell
- The Partners Healthcare International Harvard Medical School Boston MA
| | - Gloria J. Kuhn
- The Wayne State University School of Medicine Detroit MI
| | - Wendy C. Coates
- Harbor–UCLA Medical Center University of California at Los Angeles Los Angeles CA
| | | | - Jonathan Fisher
- Beth Israel Deaconess Medical Center Harvard Medical School Boston MA
| | | | | |
Collapse
|