The need for outcome measures in medical education

A preliminary study of the probitive value of personality assessment in medical school admissions within the United States

BACKGROUND

Allopathic medicine faces a daunting challenge of selecting the best applicants because of the very high applicant / matriculant ratio. The quality of graduates ultimately reflects the quality of medical practice. Alarming recent trends in physician burnout, misconduct and suicide raise questions of whether we are selecting the right candidates. The United States (US) lags far behind the United Kingdom (UK) and Europe in the study of non-cognitive tests in medical school admissions. Although more recently, medical schools in both the UK, Europe and the US have begun to use situational judgement tests such as the Computer-Based Assessment for Sampling Personal Characteristics (CASPer) and the situational judgement test (SJT), recently developed by the Association of American Medical Colleges (AAMC) and that these tests are, in a sense non-cognitive in nature, direct personality tests per se have not been utilized. We have historically used, in the admissions process within the US, knowledge, reasoning and exam performance, all of which are largely influenced by intelligence and also improved with practice. Personality, though also undoubtedly influenced by intelligence, is fundamentally different and subject to different kinds of measurements.

METHODS

A popular personality measurement used over the past two decades within the US in business and industry, but not medical school has been the Neo Personality Inventory - Revised (NEO-PI-R) Test. This test has not been utilized regularly in allopathic medicine probably because of the paucity of exploratory retrospective and validating prospective studies. The hypothesis which we tested was whether NEO-PI-R traits exhibited consistency between two institutions and whether their measurements showed probative value in predicting academic performance.

RESULTS

Our retrospective findings indicated both interinstitutional consistencies and both positive and negative predictive values for certain traits whose correlative strengths exceeded traditional premed metrics: medical college admission test (MCAT) scores, grade point average (GPA), etc. for early academic performance.

CONCLUSIONS

Our exploratory studies should catalyze larger and more detailed confirmatory studies designed to validate the importance of personality traits not only in predicting early medical school performance but also later performance in one's overall medical career.

Evaluating the effectiveness of undergraduate clinical education programs

Medical schools should use a variety of measures to evaluate the effectiveness of their clinical curricula. Both outcome measures and process measures should be included, and these can be organized according to the four-level training evaluation model developed by Donald Kirkpatrick. Managing evaluation data requires the institution to employ deliberate strategies to monitor signals in real-time and aggregate data so that informed decisions can be made. Future steps in program evaluation includes increased emphasis on patient outcomes and multi-source feedback, as well as better integration of existing data sources.

Digital Education in Health Professions: The Need for Overarching Evidence Synthesis

Synthesizing evidence from randomized controlled trials of digital health education poses some challenges. These include a lack of clear categorization of digital health education in the literature; constantly evolving concepts, pedagogies, or theories; and a multitude of methods, features, technologies, or delivery settings. The Digital Health Education Collaboration was established to evaluate the evidence on digital education in health professions; inform policymakers, educators, and students; and ultimately, change the way in which these professionals learn and are taught. The aim of this paper is to present the overarching methodology that we use to synthesize evidence across our digital health education reviews and to discuss challenges related to the process. For our research, we followed Cochrane recommendations for the conduct of systematic reviews; all reviews are reported according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidance. This included assembling experts in various digital health education fields; identifying gaps in the evidence base; formulating focused research questions, aims, and outcome measures; choosing appropriate search terms and databases; defining inclusion and exclusion criteria; running the searches jointly with librarians and information specialists; managing abstracts; retrieving full-text versions of papers; extracting and storing large datasets, critically appraising the quality of studies; analyzing data; discussing findings; drawing meaningful conclusions; and drafting research papers. The approach used for synthesizing evidence from digital health education trials is commonly regarded as the most rigorous benchmark for conducting systematic reviews. Although we acknowledge the presence of certain biases ingrained in the process, we have clearly highlighted and minimized those biases by strictly adhering to scientific rigor, methodological integrity, and standard operating procedures. This paper will be a valuable asset for researchers and methodologists undertaking systematic reviews in digital health education.

Objectively structured verbal examination to assess surgical clerkship education: An evaluation of students' perception

Objective

The objectivity and reliability of examining methods are controversial. We subjected fourth-year medical students to a specially designed verbal exam which we called objectively structured verbal examination. We aimed to evaluate feedback from students about objectively structured verbal examination as an assessment instrument for gauging their surgical knowledge.

Material and Methods

Objectively structured verbal examination modules were developed according to the learning goals of the surgical clerkship. Upon finishing surgery rotation, the students were subjected to objectively structured verbal examination as part of their final evaluation. The students' perception of objectively structured verbal examination was assessed by their responses to a questionnaire.

Results

Forty-two of 58 students returned filled questionnaires. Objectively structured verbal examination was accepted by 72% of the students as an objective tool, and 86% of them found it enabled unbiased evaluation. Overall, most students expressed positive feedback regarding objectively structured verbal examination.

Conclusion

The feedback received from students showed that objectively structured verbal examination is a reliable and objective method to assess their knowledge. This feedback reflects that objectively structured verbal examination merits further development and enhancement.

Do personality traits assessed on medical school admission predict exit performance? A UK-wide longitudinal cohort study

Traditional methods of assessing personality traits in medical school selection have been heavily criticised. To address this at the point of selection, "non-cognitive" tests were included in the UK Clinical Aptitude Test, the most widely-used aptitude test in UK medical education (UKCAT: http://www.ukcat.ac.uk/ ). We examined the predictive validity of these non-cognitive traits with performance during and on exit from medical school. We sampled all students graduating in 2013 from the 30 UKCAT consortium medical schools. Analysis included: candidate demographics, UKCAT non-cognitive scores, medical school performance data-the Educational Performance Measure (EPM) and national exit situational judgement test (SJT) outcomes. We examined the relationships between these variables and SJT and EPM scores. Multilevel modelling was used to assess the relationships adjusting for confounders. The 3343 students who had taken the UKCAT non-cognitive tests and had both EPM and SJT data were entered into the analysis. There were four types of non-cognitive test: (1) libertariancommunitarian, (2) NACE-narcissism, aloofness, confidence and empathy, (3) MEARS-self-esteem, optimism, control, self-discipline, emotional-nondefensiveness (END) and faking, (4) an abridged version of 1 and 2 combined. Multilevel regression showed that, after correcting for demographic factors, END predicted SJT and EPM decile. Aloofness and empathy in NACE were predictive of SJT score. This is the first large-scale study examining the relationship between performance on non-cognitive selection tests and medical school exit assessments. The predictive validity of these tests was limited, and the relationships revealed do not fit neatly with theoretical expectations. This study does not support their use in selection.

Predictors of professional behaviour and academic outcomes in a UK medical school: A longitudinal cohort study

BACKGROUND

Over the past 70 years, there has been a recurring debate in the literature and in the popular press about how best to select medical students. This implies that we are still not getting it right: either some students are unsuited to medicine or the graduating doctors are considered unsatisfactory, or both.

AIM

To determine whether particular variables at the point of selection might distinguish those more likely to become satisfactory professional doctors, by following a complete intake cohort of students throughout medical school and analysing all the data used for the students' selection, their performance on a range of other potential selection tests, academic and clinical assessments throughout their studies, and records of professional behaviour covering the entire five years of the course.

METHODS

A longitudinal database captured the following anonymised information for every student (n = 146) admitted in 2007 to the Hull York Medical School (HYMS) in the UK: demographic data (age, sex, citizenship); performance in each component of the selection procedure; performance in some other possible selection instruments (cognitive and non-cognitive psychometric tests); professional behaviour in tutorials and in other clinical settings; academic performance, clinical and communication skills at summative assessments throughout; professional behaviour lapses monitored routinely as part of the fitness-to-practise procedures. Correlations were sought between predictor variables and criterion variables chosen to demonstrate the full range of course outcomes from failure to complete the course to graduation with honours, and to reveal clinical and professional strengths and weaknesses.

RESULTS

Student demography was found to be an important predictor of outcomes, with females, younger students and British citizens performing better overall. The selection variable "HYMS academic score", based on prior academic performance, was a significant predictor of components of Year 4 written and Year 5 clinical examinations. Some cognitive subtest scores from the UK Clinical Aptitude Test (UKCAT) and the UKCAT total score were also significant predictors of the same components, and a unique predictor of the Year 5 written examination. A number of the non-cognitive tests were significant independent predictors of Years 4 and 5 clinical performance, and of lapses in professional behaviour. First- and second-year tutor ratings were significant predictors of all outcomes, both desirable and undesirable. Performance in Years 1 and 2 written exams did not predict performance in Year 4 but did generally predict Year 5 written and clinical performance.

CONCLUSIONS

Measures of a range of relevant selection attributes and personal qualities can predict intermediate and end of course achievements in academic, clinical and professional behaviour domains. In this study HYMS academic score, some UKCAT subtest scores and the total UKCAT score, and some non-cognitive tests completed at the outset of studies, together predicted outcomes most comprehensively. Tutor evaluation of students early in the course also identified the more and less successful students in the three domains of academic, clinical and professional performance. These results may be helpful in informing the future development of selection tools.

Does a summative portfolio foster the development of capabilities such as reflective practice and understanding ethics? An evaluation from two medical schools

Portfolios need to be evaluated to determine whether they encourage students to develop in capabilities such as reflective practice and ethical judgment. The aims of this study were (i) to determine whether preparing a portfolio helps promote students' development in a range of capabilities including understanding ethical and legal principles, reflective practice and effective communication, and (ii) to determine to what extent the format of the portfolio affected the outcome by comparing the experiences of students at two different medical schools. A questionnaire was designed to evaluate undergraduate medical students' experiences of completing a portfolio at two medical schools. A total of 526 (45% response rate) students answered the on-line questionnaire. Students from both medical schools gave the highest ranking for the portfolio as a trigger for reflective practice. 63% of students agreed their portfolio helped them develop reflective practice skills (p < 0.001), whereas only 22% disagreed. 48% of students agreed portfolios helped them understand ethical and legal principles whereas 29% disagreed (p < 0.001). In contrast, only 34% of students thought the portfolio helped them to develop effective communication. Students perceive portfolio preparation as an effective learning tool for the development of capabilities such as understanding ethical and legal principles and reflective practice, whereas other capabilities such as effective communication require complementary techniques and other modes of assessment.

Evaluating the effectiveness of curriculum change. Is there a difference between graduating student outcomes from two different curricula?

BACKGROUND

Changing a curriculum raises the question whether the results, new curriculum student outcomes, are different from old curriculum student outcomes.

AIMS

To see whether different curricula produce different outcomes, we compared test and questionnaire results of two cohorts. We wanted to know if there is a difference on knowledge and skills test results, and on the number of times students practiced their skills during their final internships.

METHOD

We used two validated test instruments: the Dutch Progress Test (PT) and the Objective Structured Clinical Examination (OSCE). For reporting their skills practice, we asked students of both cohorts to fill out a basic skills questionnaire.

RESULTS

We found no significant difference between both cohorts on the results of their knowledge test and their report on skills practice. On the OSCE, students from the new curriculum cohort scored significantly higher than old curriculum students.

CONCLUSION

Curriculum change can lead to improvements in graduating students' outcome results.

Evaluation of a learner-designed course for teaching health research skills in Ghana

BACKGROUND

In developing countries the ability to conduct locally-relevant health research and high quality education are key tools in the fight against poverty. The objective of our study was to evaluate the effectiveness of a novel UK accredited, learner-designed research skills course delivered in a teaching hospital in Ghana.

METHODS

Study participants were 15 mixed speciality health professionals from Komfo Anokye Teaching Hospital, Kumasi, Ghana. Effectiveness measures included process, content and outcome indicators to evaluate changes in learners' confidence and competence in research, and assessment of the impact of the course on changing research-related thinking and behaviour. Results were verified using two independent methods.

RESULTS

14/15 learners gained research competence assessed against UK Quality Assurance Agency criteria. After the course there was a 36% increase in the groups' positive responses to statements concerning confidence in research-related attitudes, intentions and actions. The greatest improvement (45% increase) was in learners' actions, which focused on strengthening institutional research capacity. 79% of paired before/after responses indicated positive changes in individual learners' research-related attitudes (n = 53), 81% in intention (n = 52) and 85% in action (n = 52). The course had increased learners' confidence to start and manage research, and enhanced life-long skills such as reflective practice and self-confidence. Doing their own research within the work environment, reflecting on personal research experiences and utilising peer support and pooled knowledge were critical elements that promoted learning.

CONCLUSION

Learners in Ghana were able to design and undertake a novel course that developed individual and institutional research capacity and met international standards. Learning by doing and a supportive peer community at work were critical elements in promoting learning in this environment where tutors were scarce. Our study provides a model for delivering and evaluating innovative educational interventions in developing countries to assess whether they meet external quality criteria and achieve their objectives.

Helping each other to learn--a process evaluation of peer assisted learning

BACKGROUND

The benefits of Peer Assisted Learning (PAL) are well established with positive effects on examination scores, student satisfaction and personal and professional development reported. PAL is increasingly utilised as a resource within medical education where the restrictions on resources have forced teachers to look at creating new educational environments which can be delivered at a lower cost. This study sought to evaluate the processes at work as the emphasis of PAL research to date has largely been on the consideration of student outcomes.

METHODS

Fifth-year medical undergraduates, who had completed their communication skills modular training and attended a preparatory workshop, facilitated a role-play session for their second-year colleagues within an Early Patient Contact programme. Semi-structured interviews and focus groups were used to collect data at different time points in order to establish the views of peer learners and tutors towards this new method of teaching. The data was analysed according to the principles framework analysis using N-vivo software. Themes were shared and debated with the multidisciplinary team of authors and a concordance of views on common themes was reached after discussion and debate.

RESULTS

Analysis of the data resulted in the emergence of three thematic categories: Learning Environment, Educational Exchange and Communication and Modelling. The data demonstrated a concordance of the views between peer tutors and learners on barriers and levers of this approach as well as a heightened awareness of the learning environment and the educational exchange occurring therein.

CONCLUSION

The data is significant as it not only demonstrates a high level of acceptability among tutors and learners for PAL but also indicates the reciprocity of educational exchange that appears to occur within the PAL setting. This study highlights some of the unique characteristics of PAL and we recommend the development of further qualitative studies around peer learners and tutors views of this process.