Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity

Invited review: Prevalence, risk factors, treatment, and barriers to best practice adoption for lameness and injuries in dairy cattle-A narrative review

Lameness and leg injuries are both painful and prevalent across the dairy industry, and are a major welfare concern. There has been a considerable amount of research focused on investigating the risk factors associated with lameness and injuries and how they might be prevented and treated. The objectives of this narrative review were to summarize herd-level prevalence estimates, risk factors, strategies for prevention, control, and treatment of these conditions, and the barriers to best practice adoption for lameness and injuries on dairy farms. There is a relatively high within-herd prevalence of lameness on dairy farms globally, with a recent systematic review estimating the mean prevalence at 22.8%. Similarly, there is a relatively high prevalence of hock injuries, with within-herd estimates ranging from 12% to 81% of cows affected. Knee and neck injuries have been reported to be less common, with 6% to 43% and 1% to 33%, respectively. Numerous risk factors have been associated with the incidence of lameness, notably housing (e.g., access to pasture, bedding depth, bedding type, flooring type, stall design), management (e.g., stall cleanliness, frequency of trimming, holding times, stocking density), and cow-level (e.g., body condition, parity, injured hocks) factors. Risk factors associated with hock injuries can be similarly classified into housing (e.g., bedding type and depth, outdoor access, parlor type, stall design), management (e.g., bedding depth, cleanliness), and cow (e.g., parity, days in milk, lameness) factors. Key preventative approaches for lameness include routine preventative and corrective hoof trimming, improving hoof cushioning and traction through access to pasture or adding rubber flooring, deep-bedded stalls, sand bedding, ensuring appropriate stocking densities, reduced holding times, and the frequent use of routine footbaths. Very little research has been conducted on hock, knee, and neck injury prevention and recovery. Numerous researchers have concluded that both extrinsic (e.g., time, money, space) and intrinsic (e.g., farmer attitude, perception, priorities, and mindset) barriers exist to addressing lameness and injuries on dairy farms. There are many diverse stakeholders in lameness and injury management including the farmer, farm staff, veterinarian, hoof trimmer, nutritionist, and other advisors. Addressing dairy cattle lameness and injuries must, therefore, consider the people involved, as it is these people who are influencing and implementing on-farm decisions related to lameness prevention, treatment, and control.

Development and initial validation of a video-based peroral endoscopic myotomy assessment tool

BACKGROUND AND AIMS

Video analysis has emerged as a potential strategy for performance assessment and improvement. We aimed to develop a video-based skill assessment tool for peroral endoscopic myotomy (POEM).

METHODS

POEM was deconstructed into basic procedural components through video analysis by an expert panel. A modified Delphi approach and 2 validation exercises were conducted to refine the POEM assessment tool (POEMAT). Twelve assessors used the final POEMAT version to grade 10 videos. Fully crossed generalizability (G) studies investigated the contributions of assessors, endoscopists' performance, and technical elements to reliability. G coefficients below .5 were considered unreliable, between .5 and .7 as modestly reliable, and above .7 as indicative of satisfactory reliability.

RESULTS

After task deconstruction, discussions, and the modified Delphi process, the final POEMAT comprised 9 technical elements. G analysis showed low variance for endoscopist performance (.8%-24.9%) and high interrater variability (range, 63.2%-90.1%). The G score was moderately reliable (≥.60) for "submucosal tunneling" and "myotomy" and satisfactorily reliable (≥.70) for "active hemostasis" and "mucosal closure."

CONCLUSIONS

We developed and established initial content and response process validity evidence for the POEMAT. Future steps include appraisal of the tool using a wider range of POEM videos to establish and improve the discriminative validity of this tool.

Assessing the inter-rater and intra-rater reliability of the Physical Therapy Competence Assessment for Airway Suctioning (PT-CAAS)

BACKGROUND AND PURPOSE

The Physical Therapy Competence Assessment for Airway Suctioning (PT-CAAS) is a recently developed measure to assess the clinical competence of physiotherapists who perform airway suctioning with adults. The purpose of this study was to assess the inter-rater and intra-rater reliability of the PT-CAAS.

METHODS

Scoring rules were developed through expert consultation. Reliability was then assessed using nine videos of suctioning performed in a simulated learning environment. A repeated measures design was used, with two replicate sets of measurements made by each participant for all videos. Data were analyzed using a repeated measures model for the concurrent assessment of inter-rater and intra-rater reliability. Participants were physiotherapists with suctioning experience.

RESULTS

Twenty physiotherapists completed initial scoring and re-scoring for all nine videos; their data were included in the analysis. Intraclass correlation coefficients (ICCs) for inter-rater reliability ranged from 0.569 [lower one-sided 95% confidence interval (CI): 0.395; standard error of measurement (SEM): 0.963] for infection control to 0.759 (lower one-sided 95% CI: 0.612; SEM: 0.722) for post-suctioning assessment and care. The inter-rater ICC for overall performance was 0.752 (lower one-sided 95% CI: 0.602; SEM: 0.660). ICCs for intra-rater reliability ranged from 0.759 (lower one-sided 95% CI: 0.197; SEM 0.721) for infection control to 0.860 (lower one-sided 95% CI: 0.544; SEM: 0.550) for post-suctioning assessment and care. The intra-rater ICC for overall performance was 0.867 (lower one-sided 95% CI: 0.559; SEM: 0.483).

DISCUSSION

Evidence of moderate to good inter-rater and good intra-rater reliability was found; however, the results should be interpreted with caution given the wide CIs and relatively large SEMs. Improved assessor training and assessments of reliability using a larger sample size are recommended.

Medical student coping and performance in simulated disasters

OBJECTIVES

Coping with the stress of real and simulated disasters is thought to be integral to the performance of emergency medicine providers. Yet, little is known about which coping strategies are employed in these scenarios and whether differential use of strategies predicts actual clinical and interpersonal performance.

METHODS

Thirty-four medical students were evaluated by trained simulated patients and physician observers across 111 clinical encounters during a simulated disaster. Linear Mixed Effects Modelling was used to test study hypotheses while accounting for demographic variables, psychological factors, and the dependency of multiple encounters for each participant.

RESULTS

Results indicated that multilevel modeling was necessary. Positive thinking positively predicted observed clinical performance whereas avoidant coping was a negative predictor. Anticipatory anxiety and positive affect, but not reported coping, positively predicted student interpersonal performance.

CONCLUSIONS

The present study indicates that the way medical students report managing the stress of disaster scenarios has clear links to their observed clinical performance above and beyond demographic and psychological factors. It further demonstrates the feasibility of empirically identifying specific coping strategies that may be important targets for disaster response training.

Observational Behavior Assessment for Psychological Competencies in Police Officers: A Proposed Methodology for Instrument Development

This paper proposes and showcases a methodology to develop an observational behavior assessment instrument to assess psychological competencies of police officers. We outline a step-by-step methodology for police organizations to measure and evaluate behavior in a meaningful way to assess these competencies. We illustrate the proposed methodology with a practical example. We posit that direct behavioral observation can be key in measuring the expression of psychological competence in practice, and that psychological competence in practice is what police organizations should care about. We hope this paper offers police organizations a methodology to perform scientifically informed observational behavior assessment of their police officers’ psychological competencies and inspires additional research efforts into this important area.

How Different Indicator-Dimension Ratios in Assessment Center Ratings Affect Evidence for Dimension Factors

Previous research on the construct validity of assessment center (AC) ratings has usually struggled to find support for dimension factors as an underlying source of variance of these ratings. Confirmatory factor analysis (CFA) remains the most widely used method to specify and validate the internal structure of AC ratings. However, the research support for dimension effects in AC ratings remains mixed. In addition, competing CFA models (e.g., correlated dimensions-correlated exercises models) are often plagued by non-convergence and estimation problems. Recently, it has been proposed that increasing the number of indicators per dimension and exercise combination might help to find support for dimension factors, in addition to exercise factors, in CFAs of AC ratings. Furthermore, it was also suggested that the increased ratio of indicators to dimensions may also solve some of the methodological problems associated with CFA models used to model AC ratings. However, in this research it remained unclear whether the support for dimension factors was solely due to the use of a larger indicator-dimension ratio or due to parceling that combines several behavioral indicators per dimension and exercise combination into more reliable measures of the targeted dimension. These are important empirical questions that have been left unanswered in the literature but can be potentially meaningful in seeking more balanced support for dimension effects in AC research. Using data from N = 213 participants from a 1-day AC, we aimed to investigate the impact of using different indicator-dimension ratios when specifying CFA models of AC ratings. Therefore, we investigated the impact of using different indicator-dimension ratios in the form of item parcels with data from an operational AC. On average, using three parcels eventually led to support for dimension factors in CFAs. However, exercise-based CFA models still performed better than dimension-based models. Thus, the present results point out potential limits concerning the generalizability of recent results that provided support for dimension factors in ACs.

A video anchored rating scale leads to high inter-rater reliability of inexperienced and expert raters in the absence of rater training

BACKGROUND

Our objective was to assess the impact of incorporating videos in a behaviorally anchored performance rating scale on the inter-rater reliability (IRR) of expert, intermediate and novice raters.

METHODS

The Intra-corporeal Suturing Assessment Tool (ISAT) was modified to include short video clips demonstrating poor, average, and expert performances. Blinded raters used this tool to assess videos of trainees performing suturing on a porcine model. Three attending surgeons, 4 residents, and 4 novice raters participated; no rater training was provided. The IRR was then compared among rater groups.

RESULTS

The IRR using the modified ISAT was high at 0.80 (p < 0.001). Ratings were significantly correlated with trainee objective suturing scores for all rater groups (experts: R = 0.84, residents: R = 0.81, and novices: R = 0.69; p < 0.001).

CONCLUSIONS

Incorporating video anchors (to define performance) in the ISAT led to high IRR and enabled novices to achieve similar consistency in their ratings as experts.

A novel evaluation of two related and two independent algorithms for eye movement classification during reading

Nystrӧm and Holmqvist have published a method for the classification of eye movements during reading (ONH) (Nyström & Holmqvist, 2010). When we applied this algorithm to our data, the results were not satisfactory, so we modified the algorithm (now the MNH) to better classify our data. The changes included: (1) reducing the amount of signal filtering, (2) excluding a new type of noise, (3) removing several adaptive thresholds and replacing them with fixed thresholds, (4) changing the way that the start and end of each saccade was determined, (5) employing a new algorithm for detecting PSOs, and (6) allowing a fixation period to either begin or end with noise. A new method for the evaluation of classification algorithms is presented. It was designed to provide comprehensive feedback to an algorithm developer, in a time-efficient manner, about the types and numbers of classification errors that an algorithm produces. This evaluation was conducted by three expert raters independently, across 20 randomly chosen recordings, each classified by both algorithms. The MNH made many fewer errors in determining when saccades start and end, and it also detected some fixations and saccades that the ONH did not. The MNH fails to detect very small saccades. We also evaluated two additional algorithms: the EyeLink Parser and a more current, machine-learning-based algorithm. The EyeLink Parser tended to find more saccades that ended too early than did the other methods, and we found numerous problems with the output of the machine-learning-based algorithm.

Toward a Better Understanding of Assessment Centers: A Conceptual Review

Assessment centers (ACs) are employed for selecting and developing employees and leaders. They are interpersonal at their core because they consist of interactive exercises. Minding this perspective, this review focuses on the role of the assessee, the assessor, and the AC design, as well as their interplay in the interpersonal situation of the AC. Therefore, it addresses which conceptual perspectives have increased our understanding of ACs in this context. Building on this, we review relevant empirical findings. On this basis, the review contributes to an empirically driven understanding of the interpersonal nature of ACs and provides directions for practice and future research avenues on this topic as well as on technology in ACs and cross-cultural applications.

Core principles of assessment in competency-based medical education

The meaningful assessment of competence is critical for the implementation of effective competency-based medical education (CBME). Timely ongoing assessments are needed along with comprehensive periodic reviews to ensure that trainees continue to progress. New approaches are needed to optimize the use of multiple assessors and assessments; to synthesize the data collected from multiple assessors and multiple types of assessments; to develop faculty competence in assessment; and to ensure that relationships between the givers and receivers of feedback are appropriate. This paper describes the core principles of assessment for learning and assessment of learning. It addresses several ways to ensure the effectiveness of assessment programs, including using the right combination of assessment methods and conducting careful assessor selection and training. It provides a reconceptualization of the role of psychometrics and articulates the importance of a group process in determining trainees' progress. In addition, it notes that, to reach its potential as a driver in trainee development, quality care, and patient safety, CBME requires effective information management and documentation as well as ongoing consideration of ways to improve the assessment system.

Ethical challenges in assessment centres in South Africa

Orientation: Assessment Centres (ACs) are used globally for the selection and development of candidates. Limited empirical evidence exists of the ethical challenges encountered in the use of ACs, especially in South Africa (SA).Research purpose: Firstly, to explore possible ethical challenges related to ACs in SA from the vantage point of the practitioner and, secondly, to search for possible solutions to these.Motivation for the study: Decisions based on AC outcomes have profound implications for participants and organisations, and it is essential to understand potential ethical challenges to minimise these, specifically in the SA context, given its socio-political history, multiculturalism, diversity and pertinent legal considerations.Research design, approach and method: A qualitative, interpretative research design was chosen. Data were collected by means of a semi-structured survey that was completed by 96 AC practitioners who attended an AC conference. Content analysis and thematic interpretation were used to make sense of the data. The preliminary findings were assessed by a focus group of purposively selected subject-matter experts (n = 16) who provided informed insights, which were incorporated into the final findings. The focus group suggested ways in which specific ethical challenges may be addressed.Main findings: The findings revealed many ethical challenges that can be better understood within a broad framework encompassing 10 themes: Universal ethical values; multicultural global contexts; the regulatory-legal framework for ACs in SA; characteristics of the assessor; psychometric properties of the AC; characteristics of the participant; bias and prejudice; governance of the AC process; ethical culture of the employer organisation and the evasive nature of ethics as a concept.Practical and managerial implications: Considerable risk exists for the unethical use of ACs. An awareness of possible areas of risk may assist AC stakeholders in their search for ethical AC use.Contribution or value-add: The study may contribute to an evidence-based understanding of the ethical aspects of ACs. The recommendations may also benefit all AC stakeholders who wish to use ACs ethically.

Retention of Assessment Center Rater Training

Abstract. The purpose of this research was to examine frame-of-reference (FOR) training retention in an assessment center (AC) rater training context. In this study, we extended Gorman and Rentsch’s (2009) research showing FOR training effects on performance schemas by examining the effects immediately after training and again after a two-week nonuse period. We examined the retention effects of FOR training on performance ratings and on performance schema accuracy. The results indicated that the FOR training condition, compared to a control condition, yielded performance ratings and performance schemas more similar to expert ratings and to an expert schema, respectively. FOR training also had positive effects on ratings and performance schema accuracy assessed two weeks after training. These results support and extend the theory of FOR training, which posits that the instructed theory of performance replaces the preexisting rater schemas ( Lievens, 2001 ), and they contribute to the research on FOR training within AC contexts.

Time To Change the Bathwater: Correcting Misconceptions About Performance Ratings

Recent commentary has suggested that performance management (PM) is fundamentally “broken,” with negative feelings from managers and employees toward the process at an all-time high (Pulakos, Hanson, Arad, & Moye, 2015; Pulakos & O'Leary, 2011). In response, some high-profile organizations have decided to eliminate performance ratings altogether as a solution to the growing disenchantment. Adler et al. (2016) offer arguments both in support of and against eliminating performance ratings in organizations. Although both sides of the debate in the focal article make some strong arguments both for and against utilizing performance ratings in organizations, we believe there continue to be misunderstandings, mischaracterizations, and misinformation with respect to some of the measurement issues in PM. We offer the following commentary not to persuade readers to adopt one particular side over another but as a call to critically reconsider and reevaluate some of the assumptions underlying measurement issues in PM and to dispel some of the pervasive beliefs throughout the performance rating literature.

Rater Errors in Clinical Performance Assessments

Rater errors are some of the most significant validity threats to any performance assessment. Veterinary medical education routinely uses raters to assess student performance in a variety of scenarios (e.g., clinical assessments, OSCEs, etc.). The purpose of this "teaching tip" is to introduce veterinary medical educators to the notion of rater error, identify a list of common rater errors, and discuss how these errors can be addressed and minimized so as to produce accurate and defensible measures of student performance.

Seeing the Forest but Missing the Trees: The Role of Judgments in Performance Management

Various solutions have been proposed to “fix” performance management (PM) over the last several decades. Pulakos, Mueller Hanson, Arad, and Moye (2015) have presented a holistic approach to improving PM in organizations. Although this approach addresses several key elements related to the social context of PM, namely the buy-in of organizational stakeholders, timely and regular feedback, and future-directed feedback, we believe that several robust findings from the PM research literature could further improve this process. Are Pulakos et al. looking at the forest but missing the trees? In the following commentary, we offer several reasons that performance judgments and perhaps even informal ratings are still operating and occurring in the proposed holistic system. Therefore, advancements in other areas of PM research may offer additional ways to fix PM.

A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment

CONTEXT

The relative advantages and disadvantages of checklists and global rating scales (GRSs) have long been debated. To compare the merits of these scale types, we conducted a systematic review of the validity evidence for checklists and GRSs in the context of simulation-based assessment of health professionals.

METHODS

We conducted a systematic review of multiple databases including MEDLINE, EMBASE and Scopus to February 2013. We selected studies that used both a GRS and checklist in the simulation-based assessment of health professionals. Reviewers working in duplicate evaluated five domains of validity evidence, including correlation between scales and reliability. We collected information about raters, instrument characteristics, assessment context, and task. We pooled reliability and correlation coefficients using random-effects meta-analysis.

RESULTS

We found 45 studies that used a checklist and GRS in simulation-based assessment. All studies included physicians or physicians in training; one study also included nurse anaesthetists. Topics of assessment included open and laparoscopic surgery (n = 22), endoscopy (n = 8), resuscitation (n = 7) and anaesthesiology (n = 4). The pooled GRS-checklist correlation was 0.76 (95% confidence interval [CI] 0.69-0.81, n = 16 studies). Inter-rater reliability was similar between scales (GRS 0.78, 95% CI 0.71-0.83, n = 23; checklist 0.81, 95% CI 0.75-0.85, n = 21), whereas GRS inter-item reliabilities (0.92, 95% CI 0.84-0.95, n = 6) and inter-station reliabilities (0.80, 95% CI 0.73-0.85, n = 10) were higher than those for checklists (0.66, 95% CI 0-0.84, n = 4 and 0.69, 95% CI 0.56-0.77, n = 10, respectively). Content evidence for GRSs usually referenced previously reported instruments (n = 33), whereas content evidence for checklists usually described expert consensus (n = 26). Checklists and GRSs usually had similar evidence for relations to other variables.

CONCLUSIONS

Checklist inter-rater reliability and trainee discrimination were more favourable than suggested in earlier work, but each task requires a separate checklist. Compared with the checklist, the GRS has higher average inter-item and inter-station reliability, can be used across multiple tasks, and may better capture nuanced elements of expertise.

Why Assessment Centers Do Not Work the Way They Are Supposed To

Assessment centers (ACs) are often designed with the intent of measuring a number of dimensions as they are assessed in various exercises, but after 25 years of research, it is now clear that AC ratings that are completed at the end of each exercise (commonly known as postexercise dimension ratings) substantially reflect the effects of the exercises in which they were completed and not the dimensions they were designed to reflect. This is the crux of the long-standing “construct validity problem” for AC ratings. I review the existing research on AC construct validity and conclude that (a) contrary to previous notions, AC candidate behavior is inherently cross-situationally (i.e., cross-exercise) specific, not cross-situationally consistent as was once thought, (b) assessors rather accurately assess candidate behavior, and (c) these facts should be recognized in the redesign of ACs toward task- or role-based ACs and away from traditional dimension-based ACs.

Effects of a rater training on rating accuracy in a physical examination skills assessment

Background: The accuracy and reproducibility of medical skills assessment is generally low. Rater training has little or no effect. Our knowledge in this field, however, relies on studies involving video ratings of overall clinical performances. We hypothesised that a rater training focussing on the frame of reference could improve accuracy in grading the curricular assessment of a highly standardised physical head-to-toe examination.

Methods: Twenty-one raters assessed the performance of 242 third-year medical students. Eleven raters had been randomly assigned to undergo a brief frame-of-reference training a few days before the assessment. 218 encounters were successfully recorded on video and re-assessed independently by three additional observers. Accuracy was defined as the concordance between the raters' grade and the median of the observers' grade. After the assessment, both students and raters filled in a questionnaire about their views on the assessment.

Results: Rater training did not have a measurable influence on accuracy. However, trained raters rated significantly more stringently than untrained raters, and their overall stringency was closer to the stringency of the observers. The questionnaire indicated a higher awareness of the halo effect in the trained raters group. Although the self-assessment of the students mirrored the assessment of the raters in both groups, the students assessed by trained raters felt more discontent with their grade.

Conclusions: While training had some marginal effects, it failed to have an impact on the individual accuracy. These results in real-life encounters are consistent with previous studies on rater training using video assessments of clinical performances. The high degree of standardisation in this study was not suitable to harmonize the trained raters’ grading. The data support the notion that the process of appraising medical performance is highly individual. A frame-of-reference training as applied does not effectively adjust the physicians' judgement on medical students in real-live assessments.

Investigating the construct validity of a development assessment centre

Orientation: The assessment centre (AC) is a prominent measurement tool for selection and development.Research purpose: The aim of this study was to determine the construct validity of a one-day development assessment centre (DAC) using a convenience sample of 202 managers in a large South African banking institution.Motivation for the study: Although the AC method is popular, it has been widely criticised as to whether it predominantly measures the dimensions it is designed to measure.Research design, approach and method: The fit of the measurement models implied by the dimensions measured was analysed in a quantitative study using an ex post facto correlation design and structural equation modelling.Main findings: Bi-factor confirmatory factor analysis was used to assess the relative contribution of higher-order exercise and dimension effects. Empirical under-identification stemming from the small number of exercises designed to reflect designated latent dimensions restricted the number of DAC dimensions that could be evaluated. Ultimately, only one global dimension had enough measurement points and was analysed. The results suggested that dimension effects explained the majority of variance in the post-exercise dimension ratings.Practical/managerial implications: Candidates’ proficiency on each dimension was used as the basis for development reports. The validity of inferences holds important implications for candidates’ career development and growth.Contribution/value-add: The authors found only one study on construct validity of AC dimensions in the South African context. The present study is the first use the bi-factor approach. This study will consequently contribute to the scarce AC literature in South Africa.

Content validation: The forgotten step-child or a crucial step in assessment centre validation?

Orientation: Assessment centres (ACs) are a popular method of assessment in South Africa, as they offer a practical link to the required job, directly observed through candidate behaviour. Content is often borrowed from the USA, so research into the applicability of that content in South Africa is justified.Research purpose: This study aimed to determine whether a selected USA-developed virtual assessment centre (VAC) measured what it claims to, and to determine whether the content is suitable for South Africa.Motivation for the study: A solid pre-statistical foundation of content forms the backbone of assessing validity. Content validation analysis is well suited to analysing the relevance of AC simulations in a specific cultural context. Too often content validation is either implied, or insufficiently explained.Research design, approach and method: A content evaluation schedule was developed, consisting of 50 items covering seven content validation dimensions. Thirteen subject matter experts and nine functional experts were tasked to assess an imported VAC using this schedule.Main findings: This study provides support that the VAC appears to measure what it purports to, and that overall, the content is suitable for use in South Africa.Practical/managerial implications: Content created in the USA can be assessed for relevance and applicability for South Africa through content validation.Contribution/value-add: This study contributes to AC literature and assessment methodology by demonstrating the importance and utility of content validation. Importers and developers of AC content may use this study’s techniques to validate content to meet legislative requirements and ensure domain relevance.

Workplace-based assessment instruments in the health sciences

A historical overview of the development of assessment instruments in the health sciences is presented here, with specific attention paid to workplace-based assessment instruments. Three instruments are reviewed in detail: the mini clinical evaluation exercise (mCEX), direct observation of procedural skills (DOPS), and multi-source feedback (MSF). Features common to these instruments include their authenticity, their use in assessing professional skills, and the opportunities they afford for the provision of feedback. Although almost exclusively used in graduate medical training, they are likely to play an increasingly important role in the assessment of veterinary undergraduate students in preparation for professional practice. However, the time and cost associated with implementing these instruments raises questions about their feasibility. The continued search for the holy grail of assessment instruments and the challenges relating to the need for trained assessors leads us to conclude that ultimately, the competence of health professionals should continue to be measured using several complementary instruments.

Workplace-based assessment: effects of rater expertise

Traditional psychometric approaches towards assessment tend to focus exclusively on quantitative properties of assessment outcomes. This may limit more meaningful educational approaches towards workplace-based assessment (WBA). Cognition-based models of WBA argue that assessment outcomes are determined by cognitive processes by raters which are very similar to reasoning, judgment and decision making in professional domains such as medicine. The present study explores cognitive processes that underlie judgment and decision making by raters when observing performance in the clinical workplace. It specifically focuses on how differences in rating experience influence information processing by raters. Verbal protocol analysis was used to investigate how experienced and non-experienced raters select and use observational data to arrive at judgments and decisions about trainees' performance in the clinical workplace. Differences between experienced and non-experienced raters were assessed with respect to time spent on information analysis and representation of trainee performance; performance scores; and information processing--using qualitative-based quantitative analysis of verbal data. Results showed expert-novice differences in time needed for representation of trainee performance, depending on complexity of the rating task. Experts paid more attention to situation-specific cues in the assessment context and they generated (significantly) more interpretations and fewer literal descriptions of observed behaviors. There were no significant differences in rating scores. Overall, our findings seemed to be consistent with other findings on expertise research, supporting theories underlying cognition-based models of assessment in the clinical workplace. Implications for WBA are discussed.

MOR: a simulation-based assessment centre for evaluating the personal and interpersonal qualities of medical school candidates

CONTEXT

Medical school admissions traditionally rely heavily on cognitive variables, with non-cognitive measures assessed through interviews only. In recognition of the unsatisfactory reliability and validity of traditional interviews, medical schools are increasingly exploring alternative approaches that can provide improved measures of candidates' personal and interpersonal qualities.

METHODS

An innovative assessment centre (MOR [Hebrew acronym for 'selection for medicine']) was designed to measure candidates' personal and interpersonal attributes. Three assessment tools were developed: behavioural stations, including encounters with simulated patients and group tasks; an autobiographical questionnaire, and a judgement and decision-making questionnaire. Candidates were evaluated by trained raters on four qualities: interpersonal communication; ability to handle stress; initiative and responsibility, and self-awareness.

RESULTS

In the years 2004-05, the 588 medical school candidates with the highest cognitive scores were tested; this resulted in a change of approximately 20% in the cohort of accepted students compared with previous admission criteria. Internal consistency ranged from 0.80 to 0.88; inter-rater reliability ranged from 0.62 to 0.77 for the behavioural stations and from 0.72 to 0.95 for the questionnaires; test-retest score correlation was 0.7. The correlation between candidates' MOR scores and cognitive scores approached zero, reflecting the value of MOR in the screening process. Feedback from participants indicated that MOR was perceived as fair and appropriate for medical school screening.

DISCUSSION

MOR is a reliable tool for measuring non-cognitive attributes in medical school candidates. It has high content and face validity. Furthermore, its implementation conveys the importance of maintaining humanist characteristics in the medical profession to students and faculty staff.

Personnel Selection

We review developments in personnel selection since the previous review by Hough & Oswald (2000) in the Annual Review of Psychology. We organize the review around a taxonomic structure of possible bases for improved selection, which includes (a) better understanding of the criterion domain and criterion measurement, (b) improved measurement of existing predictor methods or constructs, (c) identification and measurement of new predictor methods or constructs, (d) improved identification of features that moderate or mediate predictor-criterion relationships, (e) clearer understanding of the relationship between predictors or between predictors and criteria (e.g., via meta-analytic synthesis), (f) identification and prediction of new outcome variables, (g) improved ability to determine how well we predict the outcomes of interest, (h) improved understanding of subgroup differences, fairness, bias, and the legal defensibility, (i) improved administrative ease with which selection systems can be used, (j) improved insight into applicant reactions, and (k) improved decision-maker acceptance of selection systems.

Assessment Center-Praxis in deutschen Großunternehmen

Zusammenfassung. In einer Befragung deutscher Großunternehmen wird untersucht, inwieweit wissenschaftliche Qualitätskriterien der Assessment Center (AC)-Methode in der Praxis umgesetzt werden. Die Kriterien beziehen sich sowohl auf die AC-Konstruktion (Anforderungsanalysen, quantitatives Verhältnis zwischen Beobachtern und Kandidaten, Schulung der Beobachter etc.) als auch auf die eingesetzten Übungen sowie die Durchführung der Assessment Center (z. B. Einsatz von verhaltensverankerten Skalen, Beobachterrotation, Evaluation etc.). Im Hinblick auf die meisten Kriterien ergibt sich ein primär positives Bild. Allerdings lassen sich auch deutliche Defizite belegen. Die Ergebnisse werden im Hinblick auf mögliche Ursachen für die mangelnde Umsetzung wissenschaftlicher Standards in der Praxis diskutiert.

Credentialing and certifying with simulation

Assessment and evaluation are integral parts of any educational and training process, and students at all levels of training respond by studying more seriously for the parts of the course or training that are assessed. To promote and enhance effective learning successfully, simulation and other teaching methods should be both formative and summative, because the ultimate goal is to ensure professional competence. This article describes a model of medical competence, and focuses on the use of medical simulation in assessment and evaluation of different levels of clinical competence using examples from experience.

Broadening perspectives on clinical performance assessment: rethinking the nature of in-training assessment

CONTEXT

In-training assessment (ITA), defined as multiple assessments of performance in the setting of day-to-day practice, is an invaluable tool in assessment programmes which aim to assess professional competence in a comprehensive and valid way. Research on clinical performance ratings, however, consistently shows weaknesses concerning accuracy, reliability and validity. Attempts to improve the psychometric characteristics of ITA focusing on standardisation and objectivity of measurement thus far result in limited improvement of ITA-practices.

PURPOSE

The aim of the paper is to demonstrate that the psychometric framework may limit more meaningful educational approaches to performance assessment, because it does not take into account key issues in the mechanics of the assessment process. Based on insights from other disciplines, we propose an approach to ITA that takes a constructivist, social-psychological perspective and integrates elements of theories of cognition, motivation and decision making. A central assumption in the proposed framework is that performance assessment is a judgment and decision making process, in which rating outcomes are influenced by interactions between individuals and the social context in which assessment occurs.

DISCUSSION

The issues raised in the article and the proposed assessment framework bring forward a number of implications for current performance assessment practice. It is argued that focusing on the context of performance assessment may be more effective in improving ITA practices than focusing strictly on raters and rating instruments. Furthermore, the constructivist approach towards assessment has important implications for assessment procedures as well as the evaluation of assessment quality. Finally, it is argued that further research into performance assessment should contribute towards a better understanding of the factors that influence rating outcomes, such as rater motivation, assessment procedures and other contextual variables.