A Systematic Review of Metrics Utilized in the Selection and Prediction of Future Performance of Residents in the United States

Background Aligning resident and training program attributes is critical. Many programs screen and select residents using assessment tools not grounded in available evidence. This can introduce bias and inappropriate trainee recruitment. Prior reviews of this literature did not include the important lens of diversity, equity, and inclusion (DEI). Objective This study's objective is to summarize the evidence linking elements in the Electronic Residency Application Service (ERAS) application with selection and training outcomes, including DEI factors. Methods A systematic review was conducted on March 30, 2022, concordant with PRISMA guidelines, to identify the data supporting the use of elements contained in ERAS and interviews for residency training programs in the United States. Studies were coded into the topics of research, awards, United States Medical Licensing Examination (USMLE) scores, personal statement, letters of recommendation, medical school transcripts, work and volunteer experiences, medical school demographics, DEI, and presence of additional degrees, as well as the interview. Results The 2599 identified unique studies were reviewed by 2 authors with conflicts adjudicated by a third. Ultimately, 231 meeting inclusion criteria were included (kappa=0.53). Conclusions Based on the studies reviewed, low-quality research supports use of the interview, Medical Student Performance Evaluation, personal statement, research productivity, prior experience, and letters of recommendation in resident selection, while USMLE scores, grades, national ranking, attainment of additional degrees, and receipt of awards should have a limited role in this process.

Impact of a Regional Grant Program Through the Lens of Social Cognitive Career Theory: A Mixed-Method Evaluation

PURPOSE

Evaluations of educational grant programs have focused on research productivity, with few examining impacts on grantees or effective program characteristics. This evaluation examined the regional grant program sponsored by Group on Educational Affairs to examine if and how grantees' careers were affected by funding, and if these experiences aligned with program goals.

METHOD

In this concurrent, mixed-methods theory-driven evaluation, quantitative and qualitative data were analyzed independently and then integrated to examine complementarity. Quantitative data examined differences among 4 geographic regions and included proposal and grantee characteristics abstracted from administrative records of 52 funded proposals from 2010-2015 grant cycles. Qualitative data from 23 interviews conducted from 2018 to 2019 explored the impact on grantees, with Social Cognitive Career Theory (SCCT) serving as a framework for deductive thematic analysis. To facilitate integration of findings, quantitative data were layered onto each interview to permit exploration of associations between the 2 data types.

RESULTS

Although significant regional differences existed in project length and amount of funding, there were few regional differences in grantee experiences. Despite small funding amounts, grants were perceived as career launching pads. The SCCT framework accounted for grantee experiences, including researcher identity formation and subsequent research, but did not capture collaboration phenomena. Integration of the 2 data types identified experience patterns unique to different groups of grantees (e.g., more or less research experience). The diversity among grantees suggests that clarification of program goals and stronger alignment with criteria for funding may be warranted.

CONCLUSIONS

This evaluation illuminates why small educational grant programs may or may not impact interest and productivity in research. Implications exist for funders, including clarifying program goals and providing support for less experienced grantees. Future research should explore grantee subsets (e.g., underrepresented in medicine) to further identify what fosters or inhibits careers of medical education scholars.

"It Changed How I Think"-Impact of Programmatic Assessment Upon Practicing Physicians: A Qualitative Study

Programmatic assessment is a systematic approach used to document and assess learner performance. It offers learners frequent formative feedback from a variety of contexts and uses both high- and low-stakes assessments to determine student progress. Existing research has explored learner and faculty perceptions of programmatic assessment, reporting favorable impact on faculty understanding of the importance of assessment stakes and feedback to learners while students report the ability to establish and navigate towards goals and reflect on their performance. The Cleveland Clinic Lerner College of Medicine (CCLCM) of Case Western Reserve University adopted programmatic assessment methods at its inception. With more than 18 years' experience with programmatic assessment and a portfolio-based assessment system, CCLCM is well-positioned to explore its graduates' perceptions of their programmatic assessment experiences during and after medical school. In 2020, the investigators interviewed 26 of the 339 physician graduates. Participants were purposefully sampled to represent multiple class cohorts (2009-2019), clinical specialties, and practice locations. The investigators analyzed interview transcripts using thematic analysis informed by the frameworks of self-determination theory and professional identity formation. The authors identified themes and support each with participant quotes from the interviews. Based on findings, the investigators compiled a series of recommendations for other institutions who have already or plan to incorporate elements of programmatic assessment into their curricula. The authors concluded by discussing future directions for research and additional avenues of inquiry.

What Is an Innovation Article? A Systematic Overview of Innovation in Health Professions Education Journals

PURPOSE

Innovation articles have their own submission category and guidelines in health professions education (HPE) journals, which suggests innovation might be a unique genre of scholarship. Yet, the requirements for innovation submissions vary among journals, suggesting ambiguity about the core content of this type of scholarship. To reduce this ambiguity, the researchers conducted a systematic overview to identify key features of innovation articles and evaluate their consistency in use across journals. Findings from this review may have implications for further development of innovation scholarship within HPE.

METHOD

In this systematic overview, conducted in 2020, the researchers identified 13 HPE journals with innovation-type articles and used content analysis to identify key features from author guidelines and publications describing what editors look for in innovation articles. The researchers then audited a sample of 39 innovation articles (3/journal) published in 2019 to determine presence and consistency of 12 innovation features within and across HPE journals. Audit findings informed the researchers' evaluation of innovation as a genre in HPE.

RESULTS

Findings show variability of innovation feature presence within and across journals. On average, articles included 7.8 of the 12 innovation features (SD 2.1, range 3-11). The most common features were description of: how the innovation was implemented (92%), a problem (90%), what was new or novel (79%), and data or outcomes (77%). On average, 5.5 (SD 1.5) out of 12 innovation features were consistently used in articles within each journal.

CONCLUSIONS

The authors identified common features of innovation article types based on journal guidelines, but there was variability in presence and consistency of these features, suggesting HPE innovations are in an emerging state of genre development. The authors discuss potential reasons for variability within this article type and highlight the need for further discussion among authors, editors, and reviewers to improve clarity.

Fundamentals of Scholarly Peer Review: A Workshop for Health Professions Educators on Practicing Scholarly Citizenship

Introduction

Education scholarship requires peer reviewers. For novice scholars, reviewing is an important developmental activity that cultivates deeper participation in the scholarship community. Yet getting started with reviewing is challenging for those not involved with the educational scholarship community. Beyond scientific expertise, reviewers must have a mentoring mindset, skills in providing constructive feedback, and knowledge of common ethical challenges associated with producing and evaluating scholarship.

Methods

Our workshop introduced novice health professions educators to peer reviewing. It included four stimulus presentations about the peer reviewer's mindset and skills, followed by reinforcement activities. The workshop could be adapted to variously sized groups. An 8:1 ratio of participants to facilitators was ideal for activities. Topics covered included considerations before accepting an invitation, the review process, the good citizen approach to reviewing, and reviewer ethics. The session concluded with suggestions for continued development of peer reviewer competencies. The workshop was evaluated using a custom survey.

Results

Throughout 2019 and early 2020, 58 health professions educators and trainees participated in the workshop across varied venues. Evaluations were obtained from 33 participants (57%). Nearly all rated the workshop as high quality and valuable to peer reviewer preparation. Most (26 of 33; 75%) gained confidence about their qualifications to serve as reviewers. Eighty percent (28 of 33) believed they could recognize ethical dilemmas.

Discussion

This workshop provided a springboard for peer reviewing health professions education scholarship. Participants generally praised the experience for introducing them to the world of peer review and preparing them for it.

"What do you think about nephrology?" A national survey of internal medicine residents

Background

Interest in nephrology has been declining among internal medicine residents but the reasons behind this observation are not well characterized. Our objective was to evaluate factors influencing residents’ choice of subspecialty.

Methods

This is a mixed-method QUAL-QUAN design study that used the results of our previously published qualitative analysis on residents’ perception of nephrology to create and pilot a questionnaire of 60 questions. The final questionnaire was distributed to 26 programs across the United States and a total of 1992 residents. We calculated response rates and tabulated participant characteristics and percentage of participant responses. We categorized choice of fellowship into 2 medical categories (Highly Sought After vs. Less Sought After) and fitted a logistic regression model of choosing a highly vs. less sought after fellowship.

Results

Four hundred fifteen out of 1992 (21%) US residents responded to the survey. Of the 268 residents planning to pursue fellowship training, 67 (25%) selected a less sought after fellowship. Female sex was associated with significantly higher odds of selecting a less sought after fellowship (OR = 2.64, 95% CI: 1.47, 4.74). Major factors deterring residents from pursuing nephrology were perception of inadequate financial compensation, broad scope of clinical practice and complexity of patient population. We observed a decline in exposure to nephrology during the clinical years of medical school with only 35.4% of respondents rotating in nephrology versus 76.8% in residency. The quality of nephrology education was rated less positively during clinical medical school years (median of 50 on a 0–100 point scale) compared to the pre-clinical years (median 60) and residency (median 75).

Conclusion

Our study attempts to explain the declining interest in nephrology. Results suggest potential targets for improvement: diversified trainee exposure, sub-specialization of nephrology, and increased involvement of nephrologists in the education of trainees.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12882-021-02397-9.

Effectiveness of a Leadership Development Course for Chief Residents: A Longitudinal Evaluation

BACKGROUND

Nonclinical skills (eg, self-regulation, team leadership, conflict resolution) are essential for success as a chief resident (CR). The literature on programs teaching these skills reports few if any effectiveness outcomes.

OBJECTIVE

We reported the outcomes of a leadership course for CRs using participants' self-reported outcomes and assessments from their program directors (PDs).

METHODS

A 2-day curriculum focused on emotional intelligence competencies, including self-awareness, self-management, social awareness, and relationship management. We used a logic model to align 2017-2018 curriculum with targeted outcomes. Questionnaires before and after the course assessed short-term and intermediate outcomes for the participants and PD interviews evaluated observed changes in CRs' performance attributable to the course.

RESULTS

A total of 74 residents participated in the course, and 65% and 59% responded to the post-course and follow-up questionnaires, respectively. Over 95% of respondents indicated developing leadership knowledge and skills and connecting with new CRs in the post-course questionnaire. During follow-up, CRs reported applying concepts learned during chief residency, using tools to address conflict, engaging in quality and patient safety projects, and continuing to interact with other participants. The relationships between reported outcomes and participants' gender/prior leadership training were not significant (P > .05), with small to medium effect sizes (0.01-0.32). All 14 PDs offered positive appraisal of the CRs, but we could not specifically attribute this growth to the course.

CONCLUSIONS

Participation in this CR leadership development course was associated with enhancement and application of leadership competencies in immediate and intermediate time frames.

Best practice versus actual practice: an audit of survey pretesting practices reported in a sample of medical education journals

Background: Despite recommendations from survey scientists, surveys appear to be utilized in medical education without the critical step of pretesting prior to survey launch. Pretesting helps ensure respondents understand questions as survey developers intended and that items and response options are relevant to respondents and adequately address constructs, topics, issues or problems. While psychometric testing is important in assessing aspects of question quality and item performance, it cannot discern how respondents, based upon their lived experiences, interpret the questions we pose.Aim: This audit study explored whether authors of medical education journal articles within audited journals reported pretesting survey instruments during survey development, as recommended by survey scientists and established guidelines/standards for survey instrument development.Methods: Five national and international medical education journals publishing survey articles from Jan. 2014 - Dec. 2015 were audited to determine whether authors reported pretesting during survey development. All abstracts within all issues of these journals were initially reviewed. Two hundred fifty-one articles met inclusion criteria using a protocol piloted and revised prior to use.Results: The number of survey articles published per journal ranged from 11 to 106. Of 251 audited articles, 181 (72.11%) described using a new instrument without pretesting, while 17 (6.77%) described using a new instrument where items were pretested. Fifty-three (21.12%) articles described using pre-existing instruments; of these, no articles (0%) reported pretesting existing survey instruments prior to use.Conclusions: Findings from this audit study indicate that reported survey pretesting appears to be lower than that reported in healthcare journals. This is concerning, as results of survey studies and evaluation projects are used to inform educational practices, guide future research, and influence policy and program development. Findings apply to both survey developers and faculty across a range of fields, including evaluation and medical education research.

Medical student resilience strategies: A content analysis of medical students' portfolios

INTRODUCTION

Stress and burnout among medical students is a well-recognized concern. A student's ability to employ resilience strategies to self-regulate behaviour is critical to the student's future career as a physician.

METHODS

We retrospectively reviewed a sampling of year 1, 2 and 5 portfolio essays focused on the Personal Development competency and performance milestones, written by 49 students from three different classes in a 5-year programme devoted to training physician investigators. Two medical educators used a framework established by Jensen and colleagues (2008) to identify the nature and prevalence of various resilience strategies (valuing the physician role, self-awareness, personal arena, professional arena, professional support and personal support) medical students reported in portfolio essays.

RESULTS

All students documented at least one strategy in their essays each year. In all years, the most commonly documented strategies were in the personal arena (95.7% of year 1, 98% of year 2 and 87.8% of year 5 portfolios). The least frequently documented strategy in all years was professional support (42.8% of year 1, 38.8% of year 2, and 28.6% of year 5 portfolios). Year 5 portfolios discussed personal support strategies (79.6%) more frequently than year 1 (53.1%) and year 2 (59.2%) portfolios.

DISCUSSION

The results suggest that medical students can identify stressors and articulate resilience strategies that can be employed to potentially address them.

The Learning Environment Counts: Longitudinal Qualitative Analysis of Study Strategies Adopted by First-Year Medical Students in a Competency-Based Educational Program

PURPOSE

The move toward competency-based education will require medical schools and postgraduate training programs to restructure learning environments to motivate trainees to take personal ownership for learning. This qualitative study explores how medical students select and implement study strategies while enrolled in a unique, nontraditional program that emphasizes reflection on performance and competence rather than relying on high-stakes examinations or grades to motivate students to learn and excel.

METHOD

Fourteen first-year medical students volunteered to participate in three, 45-minute interviews (42 overall) scheduled three months apart during 2013-2014. Two medical educators used structured interview guides to solicit students' previous assessment experiences, preferred learning strategies, and performance monitoring processes. Interviews were digitally recorded and transcribed verbatim. Participants confirmed accuracy of transcripts. Researchers independently read transcripts and met regularly to discuss transcripts and judge when themes achieved saturation.

RESULTS

Medical students can adopt an assessment for learning mind-set with faculty guidance and implement appropriate study strategies for mastery-learning demands. Though students developed new strategies at different rates during the year, they all eventually identified study and performance monitoring strategies to meet learning needs. Students who had diverse learning experiences in college embraced mastery-based study strategies sooner than peers after recognizing that the learning environment did not reward performance-based strategies.

CONCLUSIONS

Medical students can take ownership for their learning and implement specific strategies to regulate behavior when learning environments contain building blocks emphasized in self-determination theory. Findings should generalize to educational programs seeking strategies to design learning environments that promote self-regulated learning.

Psychometric Properties of an Instrument Designed to Measure the Educational Quality of Graduate Training Programs

To assess the quality of residency education programs at an academic medical center for purposes of enhancing individual graduate medical education programs, we asked residents and fellows (N = 419) to evaluate their training programs using a Web-based questionnaire (response rate = 70%). Kruskal-Wallis tests, factor analysis, correlations, generalizability/decision studies, and mean plots were used to examine trainee responses and to assess the questionnaire’s measurement properties. Exploratory factor analysis indicated that the instrument had a threefactor structure that correlated highly with overall program rating. Cronbach’s alpha exceeded .80 for all factors, and decision studies revealed that 13 to 23 raters were needed to obtain G-coefficients greater than .70. Mean plots showed that the instrument could discriminate within and among training programs at the item level and the factor level, which should help target improvements across graduate training programs within large institutions.

Time to Loosen the Apron Strings: Cohort-based Evaluation of a Learner-driven Remediation Model at One Medical School

BACKGROUND

Remediation in the era of competency-based assessment demands a model that empowers students to improve performance.

AIM

To examine a remediation model where students, rather than faculty, develop remedial plans to improve performance.

SETTING/PARTICIPANTS

Private medical school, 177 medical students.

PROGRAM DESCRIPTION

A promotion committee uses student-generated portfolios and faculty referrals to identify struggling students, and has them develop formal remediation plans with personal reflections, improvement strategies, and performance evidence. Students submit reports to document progress until formally released from remediation by the promotion committee.

PROGRAM EVALUATION

Participants included 177 students from six classes (2009-2014). Twenty-six were placed in remediation, with more referrals occurring during Years 1 or 2 (n = 20, 76 %). Unprofessional behavior represented the most common reason for referral in Years 3-5. Remedial students did not differ from classmates (n = 151) on baseline characteristics (Age, Gender, US citizenship, MCAT) or willingness to recommend their medical school to future students (p < 0.05). Two remedial students did not graduate and three did not pass USLME licensure exams on first attempt. Most remedial students (92 %) generated appropriate plans to address performance deficits.

DISCUSSION

Students can successfully design remedial interventions. This learner-driven remediation model promotes greater autonomy and reinforces self-regulated learning.

Association of research self-efficacy with medical student career interests, specialization, and scholarship: a case study

This study used variables proposed in social cognitive career theory (SCCT) to focus the evaluation of a research curriculum at the Cleveland Clinic Lerner College of Medicine of Case Western Reserve University (CCLCM). Eight cohorts of CCLCM medical students completed a web-based version of the six-scale Clinical Research Appraisal Inventory-Short Form (CRAI-SF) items at matriculation (n = 128) or graduation (n = 111) during 2009-2013. Parametric statistics were used to compare CRAI-SF scales to domains proposed in SCCT: trainees' characteristics (gender, training level, advanced degree), career interests, career intentions (medical specialty), and performance (peer-reviewed publications and required thesis topic). A number of lessons emerged in using theory to frame the evaluation of a complex educational program. Graduates rated their research self-efficacy significantly higher on all six CRAI-SF scales with large effect sizes (>.90) on five scales (Conceptualizing a Study, Study Design and Analysis, Responsible Research Conduct, Collaborating with Others, and Reporting a Study). Women and men did not have significantly different scores on CRAI-SF scales (p > .05), suggesting that the research program provides adequate supports for women students. Most thesis projects addressed clinical (36.9 %, n = 41) or translational (34.2 %, n = 38) research topics. The CRAI-SF discriminated between medical school matriculates and graduates, suggesting that research self-efficacy increases with mastery experiences. No significant relationships occurred between CRAI-SF scores and graduates' thesis topics or chosen clinical specialty. Correlations demonstrated significant relationships between graduates' perceptions of research self-efficacy and their interest in clinical research careers.

Integrating clinical medicine into biomedical graduate education to promote translational research: strategies from two new PhD programs

For several decades, a barrier has existed between research and clinical medicine, making it difficult for aspiring scientists to gain exposure to human pathophysiology and access to clinical/translational research mentors during their graduate training. In 2005, the Howard Hughes Medical Institute announced the Med Into Grad initiative to support graduate programs that integrate clinical knowledge into PhD biomedical training, with the goal of preparing a new cadre of translational researchers to work at the interface of the basic sciences and clinical medicine. Two institutions, Baylor College of Medicine and the Cleveland Clinic/Case Western Reserve University, developed new PhD programs in translational biology and/or molecular medicine. These programs teach the topics and skills that today's translational researchers must learn and expose students to clinical medicine. In this article, the authors compare and contrast the history, implementation, and evaluation of the Translational Biology and Molecular Medicine program at Baylor College of Medicine and the Molecular Medicine program at the Cleveland Clinic/Case Western Reserve University. The authors also demonstrate the feasibility of creating a multidisciplinary graduate program in molecular medicine that integrates pathophysiology and clinical medicine without extending training time. They conclude with a discussion of the similarities in training approaches that exist despite the fact that each program was independently developed and offer observations that emerged during their collaboration that may benefit others who are considering developing similar programs.

Anatomists provide the foundation for learning pathophysiology

The need for interdisciplinary graduate training programs which prepare students to conceptualize the application of their research in clinical settings continues to grow. Though several programs have been cultivated to address this need, demand still outweighs supply. The following describes a curriculum developed with the intent of incorporating medical knowledge into a PhD graduate training program. Development of this Molecular Medicine program by the Cleveland Clinic Lerner College of Medicine in collaboration with Case Western Reserve University was funded by the Howard Hughes Medical Institute "Med into Grad" initiative. The core curriculum of this program begins with a foundation in Human Physiology and Disease course in which anatomy faculty introduce students to a basic overview of gross anatomy structure and function. This course is followed by five fundamental basic science courses, a composite course focusing on principles of clinical and translation research, a course on laboratory techniques and three, 12-week research rotations. In the second year of the program, students begin their dissertation research, complete their qualifying examination, and partake in an individually tailored Clinical Experience course. Interdisciplinary graduate programs like this provide another venue for faculty in anatomical sciences to help aspiring translational scientists relate basic science knowledge to human pathophysiology and health.

Evidence within a portfolio-based assessment program: what do medical students select to document their performance?

BACKGROUND

Decisions about performance in programs of assessment that provide an array of assessment evidence require judgments about the quality of different pieces of assessment data to determine which combination of data points best represent a trainee's overall performance.

AIM

In this article, we examine the nature of evidence selected by first-year medical students to include in a portfolio used to make promotion decisions.

METHODS

We reviewed portfolios to examine the number, type, and source of assessments selected by students (n = 32) to document their performance in seven competencies. The quality of assessment data selected for each competency was rated by promotion committee members (n = 14).

RESULTS

Findings indicate that students cited multiple types and sources of available assessments. The promotion committee rated evidence quality highest for competencies where the program provided sufficient evidence for students to cite a broad range of assessments. When assessments were not provided by the program, students cited self-generated evidence.

CONCLUSION

We found that when student-constructed portfolios are part of an overall assessment system, students generally select evidence in proportion to the number and types of assessments available.

Does students' gender, citizenship, or verbal ability affect fairness of portfolio-based promotion decisions? Results from one medical school

PURPOSE

Measurement experts use four criteria to examine the fairness of tests: (1) equitable treatment for examinees, (2) equal outcomes for subgroups, (3) absence of bias, and (4) equal opportunity to learn. These criteria apply to portfolios just as they do to other assessments. This report examines the fairness of portfolio-based promotion decisions for medical students at the Cleveland Clinic Lerner College of Medicine.

METHOD

Participants were 182 first-year medical students (97 men, 85 women) from six class cohorts (2004-2009). Chi-square statistics with Yates continuity correction were used to compare overall promotion decisions to students' gender, self-reports of language fluency, and MCAT Writing Sample score. The Cramér V statistic served as an effect size index. Post hoc power analyses identified the minimum sample size to obtain acceptable power.

RESULTS

Approximately 85% of students were promoted to Year 2 of the program. Gender, U.S. citizenship, language fluency, and MCAT Writing Sample score were not significantly related to overall promotion decisions. Effect sizes were small (≤0.15) for all contingency tables, suggesting weak associations between overall promotion decisions and students' group characteristics.

CONCLUSIONS

Examining fairness, although challenging, is essential to maintain professional standards and avoid potential liability. Preliminary evidence in this study suggests that students' background characteristics and verbal abilities were not strongly related to portfolio-based promotion decisions. Schools should monitor processes that may affect fairness. This study reports on just one aspect of fairness. More research is needed to evaluate other dimensions of fairness.

How to measure success: the impact of scholarly concentrations on students--a literature review

PURPOSE

Scholarly concentrations (SCs) are elective or required curricular experiences that give students opportunities to study subjects in-depth beyond the conventional medical curriculum and require them to complete an independent scholarly project. This literature review explores the question, "What impact do SC programs have on medical students?"

METHOD

In 2008, the authors retrieved published articles using Medline, ERIC, and PsycINFO electronic databases and scanned reference lists to locate additional citations. They extracted data from selected articles using a structured form and used Kirkpatrick's evaluation model to organize learner outcomes into four categories: reactions, learning, behavior, and results.

RESULTS

Of 1,640 citations, 82 full-text papers were considered, and 39 studies met inclusion criteria. Most articles described SC programs that offered students research opportunities. Fourteen articles provided evidence that SC experiences influenced students' choice of clinical specialty or fostered their interest in research. Eight articles reported that SCs improved students' understanding of research principles and methods. Nineteen articles reported publications and presentations to document students' ability to apply acquired knowledge and skills. Twelve studies confirmed the entry of SC graduates into academic medicine with continued engagement in research or success in obtaining grant funding. Students' criticisms focused on requiring research during clinical training and the effort needed to complete scholarly projects.

CONCLUSIONS

The diversity of articles and variable results prevent definitive conclusions about the value of SCs. Findings suggest several implications for future SC program evaluations and educational research. The authors advocate increased rigor in evaluation designs to demonstrate SCs' true impact.

Evaluation of essay questions used to assess medical students' application and integration of basic and clinical science knowledge

BACKGROUND

Educators need approaches to assess medical students' abilities to apply and integrate concepts essential to medical practice.

DESCRIPTION

We used a multimethod approach to examine the quality of essay questions intended to elicit medical students' ability to apply and integrate their understanding of medical concepts.

EVALUATION

Three educators assigned essay questions (n = 120) to one of four levels of cognition. Kappa was computed before and after discussion. Faculty (n = 46) critiqued essay quality using a checklist (97% response), and students completed a questionnaire about the learning environment (91% response).

CONCLUSIONS

We identified effective approaches to evaluate the quality of essay questions and to train faculty to write essay questions of sufficient complexity. This systematic review of essay questions also encouraged review of the curriculum to determine if core concepts were being taught. It is feasible to have faculty write and critique essay questions targeted at higher levels of cognition.

Examination of a clinical teaching effectiveness instrument used for summative faculty assessment

This study explores whether a clinical teaching effectiveness (CTE) instrument provides valid scores for summative faculty assessment. The sample included all CTE instruments (n = 10,087) that learners (N = 1,194) completed to assess clinical teachers (N = 872) during 1 academic year. The authors investigated response processes (e.g., missing data, straight-line responses, level of learner), internal structure (e.g., confirmatory and exploratory factor analysis), teaching ratings by learner group (medical student or resident), and relation to other variables (e.g., correlation with global rating). Response processes identified a high prevalence of straight-line responses (same rating across all items) and differential patterns of missing data by learner group. Medical students rated their teachers higher than residents, and CTE scores had different factor structures depending on learner group. High correlation coefficients of CTE items with a single rating of overall teaching performance suggest that learners consider global performance when assessing clinical teaching performance.

Methods to assess students' acquisition, application and integration of basic science knowledge in an innovative competency-based curriculum

BACKGROUND

The Cleveland Clinic Lerner College of Medicine was designed to encourage medical students to pursue careers as physician investigators. Our faculty decided that assessment should enhance learning and adopted only formative assessments to document student performance in relation to nine broad-based competencies. No grades are used to judge student performance throughout the 5-year program. Instead, assessments are competency-based, relate directly to performance standards, and are stored in e-Portfolios to track progress and document student achievement. The class size is limited to 32 students a year.

AIMS

Schools with competency-based curricula must provide students with formative feedback to identify performance gaps and monitor progress. We describe a systematic approach to assess medical knowledge using essay-type questions (CAPPs) and multiple choice questions (SAQs) to provide medical students with weekly, formative feedback about their abilities to acquire, apply and integrate basic and clinical science concepts.

METHOD

Processes for developing performance standards, creating assessment items, training faculty, reporting student performance and monitoring outcomes are described. A case study of a Year 1 course is presented with specific examples of CAPPs and SAQs to illustrate how formative assessment data are interpreted and reported in students' e-Portfolios.

RESULTS

Preliminary evidence suggests that CAPPs and SAQs have a positive impact on students' education, a justifiable cost in light of obtained benefits and growing acceptance among stakeholders. Two student cohorts performed significantly above the population mean on USMLE Step 1, which suggests that these assessment methods have not disadvantaged students. More evidence is needed to assess the reliability and validity of these tools for formative purposes.

CONCLUSIONS

Using assessment data for formative purposes may encourage application and integration of knowledge, help students identify performance gaps, foster student development of learning plans and promote student responsibility for learning. Discussion provides applications for institutions with larger classes to consider.

Peer assessment of professional competence

BACKGROUND

Current assessment formats for medical students reliably test core knowledge and basic skills. Methods for assessing other important domains of competence, such as interpersonal skills, humanism and teamwork skills, are less well developed. This study describes the development, implementation and results of peer assessment as a measure of professional competence of medical students to be used for formative purposes.

METHODS

Year 2 medical students assessed the professional competence of their peers using an online assessment instrument. Fifteen randomly selected classmates were assigned to assess each student. The responses were analysed to determine the reliability and validity of the scores and to explore relationships between peer assessments and other assessment measures.

RESULTS

Factor analyses suggest a 2-dimensional conceptualisation of professional competence: 1 factor represents Work Habits, such as preparedness and initiative, and the other factor represents Interpersonal Habits, including respect and trustworthiness. The Work Habits factor had moderate, yet statistically significant correlations ranging from 0.21 to 0.53 with all other performance measures that were part of a comprehensive assessment of professional competence. Approximately 6 peer raters were needed to achieve a generalisability coefficient of 0.70.

CONCLUSIONS

Our findings suggest that it is possible to introduce peer assessment for formative purposes in an undergraduate medical school programme that provides multiple opportunities to interact with and observe peers.

Home care nurses' attitudes toward computers. A confirmatory factor analysis of the Stronge and Brodt instrument

Point-of-care technology for home care use will be the final step in enterprise-wide healthcare electronic communications. Successful implementation of home care point-of-care technology hinges upon nurses' attitudes toward point-of-care technology and its use in clinical practice. This study addresses the factors associated with home care nurses' attitudes using Stronge and Brodt's Nurse Attitudes Toward Computers instrument. In this study, the Nurses Attitudes Toward Computers instrument was administered to a convenience sample of 138 nurses employed by a large midwestern home care agency, with an 88% response rate. Confirmatory factor analysis corroborated the Nurses Attitudes Toward Computers' 3-dimensional factor structure for practicing nurses, which was labeled as nurses' work, security issues, and perceived barriers. Results from the confirmatory factor analysis also suggest that these 3 factors are internally correlated and represent multiple dimensions of a higher order construct labeled as nurses' attitudes toward computers. Additionally, two of these factors, nurses' work and perceived barriers, each appears to explain more variance in nurses' attitudes toward computers than security issues. Instrument reliability was high for the sample (.90), with subscale reliabilities ranging from 86 to 70.