Factor analysis methods and validity evidence: a review of instrument development across the medical education continuum

Tips for developing a coaching program in medical education

This article provides structure to developing, implementing, and evaluating a successful coaching program that effectively meets the needs of learners. We highlight the benefits of coaching in medical education and recognize that many educators desiring to build coaching programs seek resources to guide this process. We align 12 tips with Kern's Six Steps for Curriculum Development and integrate theoretical frameworks from the literature to inform the process. Our tips include defining the reasons a coaching program is needed, learning from existing programs and prior literature, conducting a needs assessment of key stakeholders, identifying and obtaining resources, developing program goals, objectives, and approach, identifying coaching tools, recruiting and training coaches, orienting learners, and evaluating program outcomes for continuous program improvement. These tips can serve as a framework for initial program development as well as iterative program improvement.

Notes to Factor Analysis Techniques for Construct Validity

This paper introduces and discusses factor analysis techniques for construct validity, including some suggestions for reporting using the evidence to support the construct validity from exploratory and confirmatory factor analysis techniques. Construct validity is a vital part of psychological testing and a prerequisite to every measurement instrument, including aptitude, achievement, and interests. Research, particularly in nursing and the health sciences, depends on reliable and valid measurements. Therefore, a growing emphasis is on assessing validity regarding the structure of test variables commonly estimated by factor analysis techniques. However, it is not always clear how to report the analysis and use it to support the construct validity. Both exploratory and confirmatory factor analysis techniques provide vital evidence to support the construct validity. However, these are not the only available evidence for construct validity, and the researcher should always consider other sources of evidence to develop and support the construct validity of their intended measures. In addition, the collection and presentation of this evidence are not limited to a time, but the validity of constructs is a continuous process that leads to validating the underlying theories from which constructs have emerged.

Cross-cultural adaptation and psychometric properties of the Persian version of the Geriatric Sleep Questionnaire (P-GSQ)

OBJECTIVE

This study aimed to validate the Geriatric Sleep Questionnaire (GSQ) for assessing subjective sleep quality among elderly individuals in Iran.

METHODS

The GSQ underwent evaluation for face and content validity. Participants were selected via convenience sampling from five healthcare centers. Sociodemographic variables, including gender, number of children, recreational activities, budget deficits, and family conflicts were analyzed. Confirmatory factor analysis was conducted to verify the results. Internal consistency was assessed using Cronbach's α, and test-retest reliability was evaluated using the intraclass correlation coefficient (ICC).

RESULTS

200 older adults (mean age 66.8 years) completed the questionnaires. Face and content validity were confirmed by 30 experts (S-CVI/average = 0.96). The final model exhibited good fit indices (χ2/df = 2.89, CFI = 0.96). The scale demonstrated acceptable internal consistency (α = 0.81) and test-retest reliability (ICC = 0.98).

CONCLUSION

The Persian GSQ demonstrates high reliability and validity for assessing sleep quality in older adults, aiding research in this field.

Investigating the internal structure of multiple mini interviews-A perspective from Pakistan

BACKGROUND

Healthcare professionals require many personal attributes in addition to cognitive abilities and psychomotor skills for competent practice. Multiple Mini- Interviews are being employed globally to assess personality attributes of candidates for selection in health professions education at all level of entry; these attributes are namely, communication skills, critical thinking, honesty, responsibility, health advocacy, empathy and sanctity of life. Considering the high stakes involved for students, faculty, institutions and the society, rigorous quality assurance mechanisms similar to those used for student assessment must be employed for student selection, throughout the continuum of medical education. It is a difficult undertaking as these psychological constructs are difficult to define and measure. Though considered to yield reliable and valid scores, studies providing multiple evidences of internal structure especially dimensionality of Multiple Mini-Interviews are sparse giving rise to questions if they are measuring a single or multiple constructs and even if they are measuring what they are purported to be measuring.

OBJECTIVE

The main objective is to provide statistical support of the multi-dimensional nature of our Multiple Mini Interviews, hypothesized a-priori, through CFA. Another objective is to provide multiple evidences for the internal structure. Our study highlights the link between content and internal structure evidences of the constructs, thus establishing that our Multiple Mini Interviews measure what they were intended to measure.

METHOD

After securing permission from the Institutional review board, an a-priori seven factor-model was hypothesized based on the attributes considered most essential for the graduating student of the institution. After operationally defining the attributes through extensive literature search, scenarios were constructed to assess them. A 5-point rating scale was used to rate each item on the station. A total 259 students participated in the multiple mini interviews over a period of three days. A training workshop had been arranged for the participating faculty.

RESULTS

The reliability coefficient using Cronbach's alpha were calculated (range from 0.73 to 0.94), Standard Error of Measurement (ranged from 0.80 to1.64), and item to station-total correlation ranged from 0.43-0.50 to 0.75-0.83. Inter-station correlation was also determined. Confirmatory factor analysis endorsed the results of Exploratory factor analysis in the study revealing a seven model fit with multiple indices of Goodness-of-fit statistics such as Root mean square error of approximation (RMSEA) value 0.05, Standardized root mean square residual (SRMR) value with less than 0.08. All these indices showed that model fit is good. The Confirmatory factor analysis confirmed the multi-dimensional nature of our MMIs and also confirmed that our stations measured the attributes that they were supposed to measure.

CONCLUSION

This study adds to the validity evidence of Multiple Mini-Interviews, in selection of candidates, with required personality traits for healthcare profession. It provides the evidence for the multi-dimensional structure of Multiple Mini interviews administered with multiple evidences for its internal structure and demonstrates the independence of different constructs being measured.

Psychometrics assessment of ethical decision-making around end-of-life care scale for adolescents in the final stage of life

Introduction

Healthcare professionals have a critical role in ethical decision-making around end-of-life care. Properly evaluating the ethical decision-making of health care professionals in end-of-life care requires reliable, tailored, and comprehensive assessments. The current study aimed to translate and assess psychometrically a Persian version of the ethical decision making in end-of-life care scale for Iranian adolescents in the final stages of life.

Methods

The present study investigates the methodology and multicenter research. 310 healthcare professionals who treat/care for adolescents at the end of life were selected from 7 cities in Iran. The original version of the end-of-life care decision-making scale was translated into Persian using the forward-backward translation method, and its psychometric properties were evaluated using COSMIN criteria.

Results

Exploratory factor analysis revealed that the factor loadings of the items ranged from 0.68 to 0.89, all of which were statistically significant. Furthermore, three factors had eigenvalues greater than 1, accounting for 81.64% of the total variance. Confirmatory factor analysis indicated a proper goodness of fit in the hypothesized factor structure. The internal consistency reliability of the tool was assessed in terms of its homogeneity, yielding a Cronbach's alpha coefficient of 0.93.

Conclusion

The Persian version of the End-of-Life Care Decision-Making Scale demonstrates satisfactory validity and reliability among healthcare professionals working with adolescents in the final stages of life. Therefore, nursing managers can utilize this tool to measure and evaluate ethical decision-making in end-of-life care for adolescents in the final stages of life and identify the most appropriate strategies, including educational interventions, to improve ethical decision-making in end-of-life care if necessary.

Psychometric Exploration of the Swedish Translation of the Sexual Orientation Microaggressions Scale (SOMS), and a Commentary on the Validity of the Construct of Microaggressions

The aim of the present study was to assess the psychometric properties of a Swedish translation of the Sexual Orientation Microaggressions Scale (SOMS) in a convenience sample of 267 Swedish LGB+ people (Mean age = 36.41). Testing suggested some strengths in terms of factor structure and 2-week test-retest reliability (ICC > .79). Also, internal consistency (α = .80-.91) and convergent validity were supported for most subscales. However, the Assumption of Deviance subscale was associated with low response variability and internal consistency (α = .35), and the correlational pattern between the Environmental Microaggressions subscale and mental health variables diverged from the overall trend. Furthermore, measurement invariance between homo- and bisexual participants was not supported for most subscales, and although microaggressions would be theoretically irrelevant to a small comparison sample of heterosexual people (N = 76, Mean age = 40.43), metric invariance of the Environmental Microaggressions subscale was supported in comparison to LGB+ people. We argue that these limitations suggest a restricted applicability of the SOMS in a Swedish context, and this has consequences for the definition and operationalization of the construct of microaggressions as a whole. Therefore, more research on the latent properties of microaggressions in Swedish as well as in other contexts is required.

Translation and psychometric assessment of a Persian version of medication safety competence scale (MSCS) for clinical nurses

Nurses play a key role in medication safety and, by extension, patient safety. Evaluation of medication safety competence in nurses requires valid, specific, and comprehensive instruments. The present study was conducted to translate and psychometric assessment a Persian version of medication safety competence scale (MSCS) for clinical nurses in Iran. This is a cross-sectional and multi-centric work of research with a methodological design. A total of 1080 clinical nurses were selected from 5 cities located in Iran. The original version of the MSCS was translated into Persian and the psychometric properties of MSCS were assessed using COSMIN criteria. The exploratory factor analysis (EFA) showed that the factor loading of the 36 items was between 0.72-0.87, all of which were significant. The confirmatory factor analysis (CFA) fitted the data well (χ²/df = 7, RMSEA = 0.01, CFI = 0.96, NFI = 0.95, and TLI = 0.97). The reliability of the instrument was assessed in terms of its internal homogeneity where the Cronbach's alpha of the whole instrument was found to be 0.96. The Persian version of MSCS for nurses possesses satisfactory validity and reliability. Thus, nurse managers can use this instrument to measure medication safety competence in nurses.

Development and Validation of a Health Behaviour Scale: Exploratory Factor Analysis on Data from a Multicentre Study in Female Primary Care Patients

Health behaviours are the most important proximal determinants of health that can be either promoting or detrimental to the health of individuals. To assess and compare health behaviours in different socioeconomic groups within the population, a comprehensive, valid, reliable, and culturally appropriate measure is needed. This study aimed to develop a health behaviour questionnaire and validate it in a sample of female patients over 45 years of age with cardiovascular disease (CVD). The development procedure encompassed the following stages: literature search and item generation, content validity testing (focus group and expert evaluation), and field testing. A preliminary 38-item Health Behaviour Scale (HBS) was developed and tested in a group of 487 female primary care patients over 45 years of age. An exploratory factor analysis (EFA) yielded a four-factor structure. Factors jointly accounted for 47% of the variance observed. The results confirmed very good internal consistency of the questionnaire. The Cronbach's alpha and McDonald's omega coefficients for the entire scale were 0.82 and 0.84, respectively. The factor and item structure of the final 16-item HBS reflects the specificity of the studied sample. This measure can be a useful tool for primary care practitioners and public health researchers by helping them to develop interventions and strategies to reinforce health-promoting behaviours.

Flipped spotters learning model: An innovative student activity-based strategy. A preparation tool for anatomy practical examinations in medical education

The flipped spotters learning model is a modern student activity-based and learner-centered method in medical education. The aim of the study was to determine if the flipped spotters learning model improves students' learning. Participants were 1214 medical students of Polish (PD) and English (ED) divisions between 2013 and 2019 academic years at the University of Warmia and Mazury in Olsztyn, Poland. They were divided into a traditional group (control group) and a flipped spotters learning group (treatment group). Each flipped spotters learning group was asked to label anatomical structures on various specimens according to the structures name list prepared by the teacher on the multiple stations. The flipped spotters learning group leaders were instructed to take pictures with the appropriately marked structures on each of the human body prosections. After completion of the class, each flipped spotters team received photos for evaluation. In the flipped spotters learning model, the students strengthened their skills and knowledge by matching specimens independently as a form of practical laboratory activities. Students' performance in gross anatomy practical examinations between the group utilizing the flipped spotters learning model, and the group with the traditional teaching model was compared. Students participating in the treatment group achieved, on average 9.9 percentage points higher among PD students, and 13.0 percentage points higher among ED students than the control group in all nine practical examinations (the effect size ranging from 0.47 to 0.95). The results suggest the positive impact of flipped spotters model on improving student's performance in the practical examinations.

Psychometric properties of a French version of the Jefferson Scale of Empathy

OBJECTIVE

To assess the reliability and construct validity of a French version of the Jefferson Scale of Empathy-Students.

METHODS

A cross-sectional study was performed among undergraduate medical students in Tunisia. A total of 833 students completed a French version of the JSE-S using convenience sampling. To identify the internal consistency aspect of the reliability, Cronbach's alpha coefficient was computed. Moreover, to assess the construct validity, the sample was randomly divided into two groups. Data from the first group (n=415) were subjected to exploratory factor analysis (EFA), with principal axing factoring (PAF) and oblimin rotation, to re-examine the underlying factor structure of the scale. Data from the second group (n=419) were used for confirmatory factor analysis (CFA) to confirm its latent variable structure. Some goodness-of-fit indices were used to assess the hypothesized model. Gender groups were compared using a t-test to check the known-group validity.

RESULTS

Reliability analysis reported an acceptable level of internal consistency, with an overall Cronbach's alpha of 0.78 (95% CI [0.75,0.80]). EFA identified a two-factor structure, accounting for 27.4% of the total variance. The two-factor model produced good fit indices when item correlated errors were considered (χ²/df = 1.95, GFI = 0.92, CFI = 0.90, PCFI = 0.79, PGFI = 0.73 and RMSEA = 0.04). Female students had a statistically significant higher empathy scores than male students (t ₍₈₃₀₎ = - 4.16, p < .001).

CONCLUSIONS

The findings support the construct validity and reliability of a French version of the JSE for medical students. This instrument appears to be useful for investigating empathy among French-speaking populations.

Dimensions of emotional distress among Brazilian workers in a COVID-19 reference hospital: A factor analytical study

BACKGROUND

The coronavirus disease 2019 (COVID-19) pandemic is an unprecedented challenge for public health and has caused the loss of millions of lives worldwide. Hospital workers play a key role in averting the collapse of the health system, but the mental health of many has deteriorated during the pandemic. Few studies have been devoted to identifying the needs of workers on frontline duty.

AIM

To investigate dimensions of common emotional symptoms and associated predictors among Brazilian workers in a COVID-19 reference hospital.

METHODS

This is an observational study of the mental health of professionals in a COVID-19 hospital in the city of São Paulo. We invited all hospital employees to respond to an online survey between July and August 2020, during the first peak of the pandemic. Data of 1000 participants who completed the survey were analyzed (83.9% were women and 34.3% were aged 30 to 40). Hospital workers self-reported the presence of symptoms of depression, anxiety, trauma-related stress, and burnout through the Patient Health Questionnaire-9, the Generalized Anxiety Disorder-7, the Impact of Event Scale-Revised and the Mini-Z Burnout Assessment respectively. Responses were assembled and subjected to exploratory factor analysis to reveal workers’ core emotional distress. Multiple linear regression models were subsequently carried out to estimate the likelihood of dimensions of distress using questions on personal motivation, threatening events, and institutional support.

RESULTS

Around one in three participants in our sample scored above the threshold of depression, anxiety, post-traumatic stress disorder, and burnout. The factor analysis revealed a three-factor structure that explained 58% of the total data variance. Core distressing emotional domains were avoidance and re-experience, depression-anxiety, and sleep changes. Regression analysis revealed that institutional support was a significant protective factor for each of these dimensions (β range = -0.41 to -0.20, P < 0.001). However, participants’ personal motivation to work in healthcare service was not associated with these emotional domains. Moreover, the likelihood of presenting the avoidance and re-experience dimension was associated with having a family member or close friend be hospitalized or die due to COVID-19 and having faced an ethical conflict.

CONCLUSION

Distressing emotional domains among hospital workers were avoidance and re-experience, depression and anxiety, and sleep changes. Improving working conditions through institutional support could protect hospital workers' mental health during devastating public health crises.

Development and Initial Validation of an Acute Readiness Monitoring Scale in Military Personnel

Personnel in many professions must remain "ready" to perform diverse activities. Managing individual and collective capability is a common concern for leadership and decision makers. Typical existing approaches for monitoring readiness involve keeping detailed records of training, health and equipment maintenance, or - less commonly - data from wearable devices that can be difficult to interpret as well as raising privacy concerns. A widely applicable, simple psychometric measure of perceived readiness would be invaluable in generating rapid evaluations of current capability directly from personnel. To develop this measure, we conducted exploratory factor analysis and confirmatory factor analysis with a sample of 770 Australian military personnel. The 32-item Acute Readiness Monitoring Scale (ARMS) demonstrated good model fit, and comprised nine factors: overall readiness; physical readiness; physical fatigue; cognitive readiness; cognitive fatigue; threat-challenge (i.e., emotional/coping) readiness; skills-and-training readiness; group-team readiness, and equipment readiness. Readiness factors were negatively correlated with recent stress, current negative affect and distress, and positively correlated with resilience, wellbeing, current positive affect and a supervisor's rating of solider readiness. The development of the ARMS facilitates a range of new research opportunities: enabling quick, simple and easily interpreted assessment of individual and group readiness.

Tools for assessment of acute psychological distress in critical illness: A scoping review

OBJECTIVES

Patients' experience of psychological distress in the intensive care unit (ICU) is associated with adverse effects, reduced satisfaction, and delayed physical and psychological recovery. There are no specific guidelines for the assessment and management of acute psychological distress during hospitalisation in the ICU. We reviewed existing tools for the assessment of acute psychological distress in ICU patients, examined evidence on their metric properties, and identified potential gaps and methodological considerations.

METHOD

A scoping review based on literature searches (Cumulative Index to Nursing and Allied Health Literature, Medical Literature Analysis and Retrieval System Online, Excerpta Medica Database, PsycINFO, Scopus, Health and Psychosocial Instruments, Dissertations and Theses Global, and Google Scholar) and predefined eligibility criteria was conducted as per current scoping review guidelines.

FINDINGS

Overall, 14 assessment tools were identified having been developed in diverse ICU settings. The identified tools assess mainly anxiety and depressive symptoms and ICU stressors, and investigators have reported various validity and reliability metrics. It was unclear whether available tools can be used in specific groups, such as noncommunicative patients and patients with delirium, brain trauma, stroke, sedation, and cognitive impairments.

CONCLUSION

Available tools have methodological limitations worth considering in future investigations. Given the high prevalence of psychiatric morbidity in ICU survivors, rigorously exploring the metric integrity of available tools used for anxiety, depressive, and psychological distress symptom assessment in the vulnerable ICU population is a practice and research priority.

RELEVANCE TO CLINICAL PRACTICE

These results have implications for the selection and implementation of psychological distress assessment methods as a means for promoting meaningful patient-centred clinical outcomes and humanising ICU care experiences.

The Simplified Nutritional Appetite Questionnaire (SNAQ) as a Screening Tool for Risk of Malnutrition: Optimal Cutoff, Factor Structure, and Validation in Healthy Community-Dwelling Older Adults

Malnutrition is an independent marker of adverse outcomes in older adults. While the Simplified Nutritional Appetite Questionnaire (SNAQ) for anorexia has been validated as a nutritional screening tool, its optimal cutoff and validity in healthy older adults is unclear. This study aims to determine the optimal cutoff for SNAQ in healthy community-dwelling older adults, and to examine its factor structure and validity. We studied 230 community-dwelling older adults (mean age 67.2 years) who were nonfrail (defined by Fatigue, Resistance, Ambulation, Illnesses & Loss (FRAIL) criteria). When compared against the risk of malnutrition using the Mini Nutritional Assessment (MNA), the optimal cutoff for SNAQ was ≤15 (area under receiver operating characteristic (ROC) curve: 0.706, sensitivity: 69.2%, specificity: 61.3%). Using exploratory factor analysis, we found a two-factor structure (Factor 1: Appetite Perception; Factor 2: Satiety and Intake) which accounted for 61.5% variance. SNAQ showed good convergent, discriminant and concurrent validity. In logistic regression adjusted for age, gender, education and MNA, SNAQ ≤15 was significantly associated with social frailty, unlike SNAQ ≤4 (odds ratio (OR) 1.99, p = 0.025 vs. OR 1.05, p = 0.890). Our study validates a higher cutoff of ≤15 to increase sensitivity of SNAQ for anorexia detection as a marker of malnutrition risk in healthy community-dwelling older adults, and explicates a novel two-factor structure which warrants further research.

Social Frailty Is Independently Associated with Mood, Nutrition, Physical Performance, and Physical Activity: Insights from a Theory-Guided Approach

Notwithstanding the increasing body of evidence that links social determinants to health outcomes, social frailty is arguably the least explored among the various dimensions of frailty. Using available items from previous studies to derive a social frailty scale as guided by the Bunt social frailty theoretical framework, we aimed to examine the association of social frailty, independently of physical frailty, with salient outcomes of mood, nutrition, physical performance, physical activity, and life–space mobility. We studied 229 community-dwelling older adults (mean age 67.22 years; 72.6% females) who were non-frail (defined by the FRAIL criteria). Using exploratory factor analysis, the resultant 8-item Social Frailty Scale (SFS-8) yielded a three-factor structure comprising social resources, social activities and financial resource, and social need fulfilment (score range: 0–8 points). Social non-frailty (SNF), social pre-frailty (SPF), and social frailty (SF) were defined based on optimal cutoffs, with corresponding prevalence of 63.8%, 28.8%, and 7.4%, respectively. In logistic regression adjusted for significant covariates and physical frailty (Modified Fried criteria), there is an association of SPF with poor physical performance and low physical activity (odds ratio, OR range: 3.10 to 6.22), and SF with depressive symptoms, malnutrition risk, poor physical performance, and low physical activity (OR range: 3.58 to 13.97) compared to SNF. There was no significant association of SPF or SF with life–space mobility. In summary, through a theory-guided approach, our study demonstrates the independent association of social frailty with a comprehensive range of intermediary health outcomes in more robust older adults. A holistic preventative approach to frailty should include upstream interventions that target social frailty to address social gradient and inequalities.

Development of the Inner City attitudinal assessment tool (ICAAT) for learners across Health care professions

BACKGROUND

Many health professions learners report feeling uncomfortable and underprepared for professional interactions with inner city populations. These learners may hold preconceptions which affect therapeutic relationships and provision of care. Few tools exist to measure learner attitudes towards these populations. This article describes the development and validity evidence behind a new tool measuring health professions learner attitudes toward inner city populations.

METHODS

Tool development consisted of four phases: 1) Item identification and generation informed by a scoping review of the literature; 2) Item refinement involving a two stage modified Delphi process with a national multidisciplinary team (n = 8), followed by evaluation of readability and response process validity with a focus group of medical and nursing students (n = 13); 3) Pilot testing with a cohort of medical and nursing students; and 4) Analysis of psychometric properties through factor analysis and reliability.

RESULTS

A 36-item online version of the Inner City Attitudinal Assessment Tool (ICAAT) was completed by 214 of 1452 undergraduate students (67.7% from medicine; 32.3% from nursing; response rate 15%). The resulting tool consists of 24 items within a three-factor model - affective, behavioural, and cognitive. Reliability (internal consistency) values using Cronbach alpha were 0.87, 0.82, and 0.82 respectively. The reliability of the whole 24-item ICAAT was 0.90.

CONCLUSIONS

The Inner City Attitudinal Assessment Tool (ICAAT) is a novel tool with evidence to support its use in assessing health care learners' attitudes towards caring for inner city populations. This tool has potential to help guide curricula in inner city health.

A general practice workplace-based assessment instrument: Content and construct validity

Introduction: Relatively few general practice (GP) workplace-based assessment instruments have been psychometrically evaluated. This study aims to establish the content validity and internal consistency of the General Practice Registrar Competency Assessment Grid (GPR-CAG).Methods: The GPR-CAG was constructed as a formative assessment instrument for Australian GP registrars (trainees). GPR-CAG items were determined by an iterative literature review, expert opinion and pilot-testing process. Validation data were collected, between 2014 and 2016, during routine clinical teaching visits within registrars' first two general practice training terms (GPT1 and GPT2) for registrars across New South Wales and the Australian Capital Territory. Factor analysis and expert consensus were used to refine items and establish GPR-CAG's internal structure. GPT1 and GPT2 competencies were analysed separately.Results: Data of 555 registrars undertaking GPT1 and 537 registrars undertaking GPT2 were included in analyses. A four-factor, 16-item solution was identified for GPT1 competencies (Cronbach's alpha range: 0.71-0.83) and a seven-factor 27-item solution for GPT2 competencies (Cronbach's alpha: 0.63-0.84). The emergent factor structures were clinically characterisable and resonant with existing medical education competency frameworks.Discussion: This study establishes initial evidence for the content validity and internal consistency of GPR-CAG. GPR-CAG appears to have utility as a formative GP training WBA instrument.

Assessing Patient Motivation for Treatment: A Systematic Review of Available Tools, Their Measurement Properties, and Conceptual Definition

BACKGROUND AND PURPOSE

Motivation is often reported by clinicians and researchers as a key factor related to treatment and health outcomes. This systematic review aims to (a) Identify and critically appraise tools that measure patient motivation for treatment and (b) determine how these tools define and evaluate motivation.

METHODS

Library databases and the search engine Google Scholar were examined. Identified tools measuring patient motivation for treatment and reported measurement properties were selected.

RESULTS

14 peer-reviewed articles covering 12 different tools made the final selection. Quality was assessed using the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) and a new measure checklist. Reliability evidence was predominantly estimated using internal consistency; validity evidence was limited, and responsiveness was seldom examined. Overall, quality ratings were poor or inadequately reported and serious methodological limitations were identified. A lack of conceptual foundation quality ratings as tools did not apply a theory related to motivation or have a clear definition of the construct of patient motivation.

CONCLUSIONS

A significant gap exists of available tools with adequate measurement properties that use relevant theoretical frameworks.

Failing to Fail nursing students among mentors: A confirmatory factor analysis of the Failing to Fail scale

AIM

The aim was to explore the psychometric properties with respect to the internal consistency reliability of the subject-specific questionnaire "Failing to Fail."

DESIGN

Cross-sectional study.

METHODS

Exploratory factor analysis with varimax rotation. A confirmatory factor analysis was used to examine the factor structure of the "Failing to Fail" scale. The sample included 336 Norwegian nurse mentors.

RESULTS

The confirmatory factor analysis confirmed a five-factor structure of the "Failing to Fail" scale with adequate model fit. The factors were named as: (a) Insufficient mentoring competence; (b) Insufficient support in the working environment; (c) Emotional process dominates the assessment; (d) Insufficient support from the university; and (e) Decision-making detached from learning outcomes. The scale proved to be feasible to test whether mentors are Failing to Fail nursing students. The confirmatory factor analysis model supported the predictive validity of the "Failing to Fail" scale.

Collecting Validity Evidence: A Hands-on Workshop for Medical Education Assessment Instruments

INTRODUCTION

There is an increasing call for developing validity evidence in medical education assessment. The literature lacks a practical resource regarding an actual development process. Our workshop teaches how to apply principles of validity evidence to existing assessment instruments and how to develop new instruments that will yield valid data.

METHODS

The literature, consensus findings of curricula and content experts, and principles of adult learning guided the content and methodology of the workshop. The workshop underwent stringent peer review prior to presentation at one international and three national academic conferences. In the interactive workshop, selected domains of validity evidence were taught with sequential cycles of didactics, demonstration, and deliberate practice with facilitated feedback. An exercise guide steered participants through a stepwise approach. Using Likert-scale items and open-response questions, an evaluation form rated the workshop's effectiveness, captured details of how learners reached the objectives, and determined participants' plans for future work.

RESULTS

The workshop demonstrated generalizability with successful implementation in diverse settings. Sixty-five learners, the majority being clinician-educators, completed evaluations. Learners rated the workshop favorably for each prompt. Qualitative comments corroborated the workshop's effectiveness. The active application and facilitated feedback components allowed learners to reflect in real time as to how they were meeting a particular objective.

DISCUSSION

This feasible and practical educational intervention fills a literature gap by showing the medical educator how to apply validity evidence to both existing and in-development assessment instruments. Thus, it holds the potential to significantly impact learner and, subsequently, patient outcomes.

A New Multisource Feedback Tool for Evaluating the Performance of Specialty-Specific Physician Groups: Validity of the Group Monitor Instrument

INTRODUCTION

Since clinical practice is a group-oriented process, it is crucial to evaluate performance on the group level. The Group Monitor (GM) is a multisource feedback tool that evaluates the performance of specialty-specific physician groups in hospital settings, as perceived by four different rater classes. In this study, we explored the validity of this tool.

METHODS

We explored three sources of validity evidence: (1) content, (2) response process, and (3) internal structure. Participants were 254 physicians, 407 staff, 621 peers, and 282 managers of 57 physician groups (in total 479 physicians) from 11 hospitals.

RESULTS

Content was supported by the fact that the items were based on a review of an existing instrument. Pilot rounds resulted in reformulation and reduction of items. Four subscales were identified for all rater classes: Medical practice, Organizational involvement, Professionalism, and Coordination. Physicians and staff had an extra subscale, Communication. However, the results of the generalizability analyses showed that variance in GM scores could mainly be explained by the specific hospital context and the physician group specialty. Optimization studies showed that for reliable GM scores, 3 to 15 evaluations were needed, depending on rater class, hospital context, and specialty.

DISCUSSION

The GM provides valid and reliable feedback on the performance of specialty-specific physician groups. When interpreting feedback, physician groups should be aware that rater classes' perceptions of their group performance are colored by the hospitals' professional culture and/or the specialty.

"The Questions Shape the Answers": Assessing the Quality of Published Survey Instruments in Health Professions Education Research

PURPOSE

Surveys are widely used in health professions education (HPE) research, yet little is known about the quality of the instruments employed. Poorly designed survey tools containing unclear or poorly formatted items can be difficult for respondents to interpret and answer, yielding low-quality data. This study assessed the quality of published survey instruments in HPE.

METHOD

In 2017, the authors performed an analysis of HPE research articles published in three high-impact journals in 2013. They included articles that employed at least one self-administered survey. They designed a coding rubric addressing five violations of established best practices for survey item design and used it to collect descriptive data on the validity and reliability evidence reported and to assess the quality of available survey items.

RESULTS

Thirty-six articles met inclusion criteria and included the instrument for coding, with one article using 2 surveys, yielding 37 unique surveys. Authors reported validity and reliability evidence for 13 (35.1%) and 8 (21.6%) surveys, respectively. Results of the item-quality assessment revealed that a substantial proportion of published survey instruments violated established best practices in the design and visual layout of Likert-type rating items. Overall, 35 (94.6%) of the 37 survey instruments analyzed contained at least one violation of best practices.

CONCLUSIONS

The majority of articles failed to report validity and reliability evidence, and a substantial proportion of the survey instruments violated established best practices in survey design. The authors suggest areas of future inquiry and provide several improvement recommendations for HPE researchers, reviewers, and journal editors.

Measuring Intent to Aid of Lay Responders: Survey Development and Validation

BACKGROUND

Increasing lay responder cardiopulmonary resuscitation and automated external defibrillator use during sudden cardiac arrest depends on an individual's choice. Investigators designed and piloted an instrument to measure the affective domain of helping behaviors by applying the theory of planned behavior (TPB) to better understand lay responders' intent to use lifesaving skills.

METHOD

Questionnaire items were compiled into 10 behavioral domains informed by the TPB constructs followed by refinement via piloting and expert review. Two samples from an American Red Cross-trained lay-responder population ( N = 4,979) provided data for an exploratory (EFA, n = 235) and confirmatory (CFA, n = 198) factor analyses. EFA derived interitem relationships into factors and affective subscales. CFA yielded statistical validation of factors and subscales.

RESULTS

The EFA identified four factors, aligned with the TPB constructs of attitudes, norms, confidence, and intention to act to explain 57% of interitem variance. The internal consistency of factor-derived subscales ranged between 0.71 and 0.91. Reduction of instrument items went from 47 to 32 (32%). The CFA yielded good model fit with the switching of the legal ramification item from the social norm to intention construct.

CONCLUSION

The Intent to Aid (I2A) survey derived from this investigation aligned with the constructs of the TPB yielding four subscales. The I2A allows health education researchers to differentiate modalities and content impact on learner intention to act in a first aid (FA) emergency. I2A compliments cognitive and psychomotor measurements of learning outcomes. The experimental instrument aims to allow curricula developers and program evaluators a means of assessing the affective domain of human learning regarding intention-to-act in an FA emergency. In combination of with assessment of functional knowledge and essential skills, this instrument may provide curricula developers and health educators an avenue to better describe intention to act in an FA emergency.

Validation of the INCEPT: A Multisource Feedback Tool for Capturing Different Perspectives on Physicians' Professional Performance

INTRODUCTION

Multisource feedback (MSF) instruments are used to and must feasibly provide reliable and valid data on physicians' performance from multiple perspectives. The "INviting Co-workers to Evaluate Physicians Tool" (INCEPT) is a multisource feedback instrument used to evaluate physicians' professional performance as perceived by peers, residents, and coworkers. In this study, we report on the validity, reliability, and feasibility of the INCEPT.

METHODS

The performance of 218 physicians was assessed by 597 peers, 344 residents, and 822 coworkers. Using explorative and confirmatory factor analyses, multilevel regression analyses between narrative and numerical feedback, item-total correlations, interscale correlations, Cronbach's α and generalizability analyses, the psychometric qualities, and feasibility of the INCEPT were investigated.

RESULTS

For all respondent groups, three factors were identified, although constructed slightly different: "professional attitude," "patient-centeredness," and "organization and (self)-management." Internal consistency was high for all constructs (Cronbach's α ≥ 0.84 and item-total correlations ≥ 0.52). Confirmatory factor analyses indicated acceptable to good fit. Further validity evidence was given by the associations between narrative and numerical feedback. For reliable total INCEPT scores, three peer, two resident and three coworker evaluations were needed; for subscale scores, evaluations of three peers, three residents and three to four coworkers were sufficient.

DISCUSSION

The INCEPT instrument provides physicians performance feedback in a valid and reliable way. The number of evaluations to establish reliable scores is achievable in a regular clinical department. When interpreting feedback, physicians should consider that respondent groups' perceptions differ as indicated by the different item clustering per performance factor.

Scholarly Concentration Program Development: A Generalizable, Data-Driven Approach

PURPOSE

Scholarly concentration programs-also known as scholarly projects, pathways, tracks, or pursuits-are increasingly common in U.S. medical schools. However, systematic, data-driven program development methods have not been described.

METHOD

The authors examined scholarly concentration programs at U.S. medical schools that U.S. News & World Report ranked as top 25 for research or primary care (n = 43 institutions), coding concentrations and mission statements. Subsequently, the authors conducted a targeted needs assessment via a student-led, institution-wide survey, eliciting learners' preferences for 10 "Pathways" (i.e., concentrations) and 30 "Topics" (i.e., potential content) augmenting core curricula at their institution. Exploratory factor analysis (EFA) and a capacity optimization algorithm characterized best institutional options for learner-focused Pathway development.

RESULTS

The authors identified scholarly concentration programs at 32 of 43 medical schools (74%), comprising 199 distinct concentrations (mean concentrations per program: 6.2, mode: 5, range: 1-16). Thematic analysis identified 10 content domains; most common were "Global/Public Health" (30 institutions; 94%) and "Clinical/Translational Research" (26 institutions; 81%). The institutional needs assessment (n = 468 medical students; response rate 60% overall, 97% among first-year students) demonstrated myriad student preferences for Pathways and Topics. EFA of Topic preferences identified eight factors, systematically related to Pathway preferences, informing content development. Capacity modeling indicated that offering six Pathways could guarantee 95% of first-year students (162/171) their first- or second-choice Pathway.

CONCLUSIONS

This study demonstrates a generalizable, data-driven approach to scholarly concentration program development that reflects student preferences and institutional strengths, while optimizing program diversity within capacity constraints.

Psychometric characterization of the obstetric communication assessment tool for medical education: a pilot study

OBJECTIVE

To characterize the psychometric properties of a novel Obstetric Communication Assessment Tool (OCAT) in a pilot study of standardized difficult OB communication scenarios appropriate for undergraduate medical evaluation.

METHODS

We developed and piloted four challenging OB Standardized Patient (SP) scenarios in a sample of twenty-one third year OB/GYN clerkship students: Religious Beliefs (RB), Angry Father (AF), Maternal Smoking (MS), and Intimate Partner Violence (IPV). Five trained Standardized Patient Reviewers (SPRs) independently scored twenty-four randomized video-recorded encounters using the OCAT. Cronbach's alpha and Intraclass Correlation Coefficient-2 (ICC-2) were used to estimate internal consistency (IC) and inter-rater reliability (IRR), respectively. Systematic variation in reviewer scoring was assessed using the Stuart-Maxwell test.

RESULTS

IC was acceptable to excellent with Cronbach's alpha values (and 95% Confidence Intervals [CI]): RB 0.91 (0.86, 0.95), AF 0.76 (0.62, 0.87), MS 0.91 (0.86, 0.95), and IPV 0.94 (0.91, 0.97). IRR was unacceptable to poor with ICC-2 values: RB 0.46 (0.40, 0.53), AF 0.48 (0.41, 0.54), MS 0.52 (0.45, 0.58), and IPV 0.67 (0.61, 0.72). Stuart-Maxwell analysis indicated systematic differences in reviewer stringency.

CONCLUSIONS

Our initial characterization of the OCAT demonstrates important issues in communications assessment. We identify scoring inconsistencies due to differences in SPR rigor that require enhanced training to improve assessment reliability. We outline a rational process for initial communication tool validation that may be useful in undergraduate curriculum development, and acknowledge that rigorous validation of OCAT training and implementation is needed to create a valuable OB communication assessment tool.

Revisiting the D-RECT tool: Validation of an instrument measuring residents' learning climate perceptions

INTRODUCTION

Credible evaluation of the learning climate requires valid and reliable instruments in order to inform quality improvement activities. Since its initial validation the Dutch Residency Educational Climate Test (D-RECT) has been increasingly used to evaluate the learning climate, yet it has not been tested in its final form and on the actual level of use - the department.

AIM

Our aim was to re-investigate the internal validity and reliability of the D-RECT at the resident and department levels.

METHODS

D-RECT evaluations collected during 2012-2013 were included. Internal validity was assessed using exploratory and confirmatory factor analyses. Reliability was assessed using generalizability theory.

RESULTS

In total, 2306 evaluations and 291 departments were included. Exploratory factor analysis showed a 9-factor structure containing 35 items: teamwork, role of specialty tutor, coaching and assessment, formal education, resident peer collaboration, work is adapted to residents' competence, patient sign-out, educational atmosphere, and accessibility of supervisors. Confirmatory factor analysis indicated acceptable to good fit. Three resident evaluations were needed to assess the overall learning climate reliably and eight residents to assess the subscales.

CONCLUSION

This study reaffirms the reliability and internal validity of the D-RECT in measuring residency training learning climate. Ongoing evaluation of the instrument remains important.

Trust me, it is valid: Research validity in pharmacy education research

Research validity is a complex concept that is often used loosely or conflated with concepts in measurement validity in published quantitative pharmacy education literature. The problem begins with a lack of clarity of the distinction between four types of research validity including measurement, statistical conclusion, internal, and external validity (i.e., generalizability). In many cases published studies provide only incomplete discussions of measurement and external validity. The problem is exacerbated within the context of measurement validity where validation efforts are often reduced to statements about established levels of reliability. Ineffective discussions of research validity make it difficult to interpret study findings. After reading this article, the reader will be able to identify the different types research validity and discuss issues of research validity in quantitative pharmacy education research more completely.

Development and psychometric validation of a novel patient survey to assess perceived quality of substance abuse treatment in South Africa

BACKGROUND

A hybrid performance measurement system that combines patient-reported outcome data with administrative data has been developed for South African substance abuse treatment services. This paper describes the development and psychometric validation of one component of this system, the South African Addiction Treatment Services Assessment (SAATSA).

METHODS

First, a national steering committee identified five domains and corresponding indicators on which treatment quality should be assessed. A decision was made to develop a patient survey to assess several of these indicators. A stakeholder work group sourced survey items and generated additional items where appropriate. The feasibility and face validity of these items were examined during cognitive response testing with 16 patients. This led to the elimination of several items. Next, we conducted an initial psychometric validation of the SAATSA with 364 patients from residential and outpatient services. Exploratory (EFA) and confirmatory factor analyses (CFA) were conducted to assess the latent structure of the SAATSA. Findings highlighted areas where the SAATSA required revision. Following revision, we conducted another psychometric validation with an additional sample of 285 patients. We used EFA and CFA to assess construct validity and we assessed reliability using Cronbach's measure of internal consistency.

RESULTS

The final version of the SAATSA comprised 31 items (rated on a four-point response scale) that correspond to six scales. Four of these scales are patient-reported outcome measures (substance use, quality of life, social connectedness and HIV risk outcomes) that together assess the perceived effectiveness of treatment. The remaining two scales assess patients' perceptions of access to and quality of care. The models for the final revised scales had good fit and the internal reliability of these scales was good to excellent, with Cronbach's α ranging from 0.72 to 0.89.

CONCLUSION

A lack of adequate measurement tools hampers efforts to improve the quality of substance abuse treatment. Our preliminary evidence suggests that the SAATSA, a novel patient survey that assesses patients' perceptions of the outcomes and quality of substance abuse treatment, is a psychometrically robust tool that can help fill this void.

Factors Influencing Seminar Learning and Academic Achievement

Many veterinary curricula use seminars, interactive educational group formats in which some 25 students discuss questions and issues relating to course themes. To get indications on how to optimize the seminar learning process for students, we aimed to investigate relationships between factors that seem to be important for the seminar learning process, and to determine how these seminar factors account for differences in students' achievement scores. A 57-item seminar evaluation (USEME) questionnaire was administered to students right after they attended a seminar. In total, 80 seminars distributed over years 1, 2, and 3 of an undergraduate veterinary medicine curriculum were sampled and 988 questionnaires were handed in. Principal factor analysis (PFA) was conducted on 410 questionnaires to examine which items could be grouped together as indicators of the same factor, and to determine correlations between the derived factors. Multilevel regression analysis was performed to explore the effects of these seminar factors and students' prior achievement scores on students' achievement scores. Within the questionnaire, four factors were identified that influence the seminar learning process: teacher performance, seminar content, student preparation, and opportunities for interaction within seminars. Strong correlations were found between teacher performance, seminar content, and group interaction. Prior achievement scores and, to a much lesser extent, the seminar factor group interaction appeared to account for differences in students' achievement scores. The factors resulting from the present study and their relation to the method of assessment should be examined further, for example, in an experimental setup.

Measuring the clinical learning environment in anaesthesia

The learning environment describes the way that trainees perceive the culture of their workplace. We audited the learning environment for trainees throughout Australia and New Zealand in the early stages of curriculum reform. A questionnaire was developed and sent electronically to a large random sample of Australian and New Zealand College of Anaesthetists trainees, with a 26% final response rate. This new instrument demonstrated good psychometric properties, with Cronbach's α ranging from 0.81 to 0.91 for each domain. The median score was equivalent to 78%, with the majority of trainees giving scores in the medium range. Introductory respondents scored their learning environment more highly than all other levels of respondents (P=0.001 for almost all comparisons). We present a simple questionnaire instrument that can be used to determine characteristics of the anaesthesia learning environment. The instrument can be used to help assess curricular change over time, alignment of the formal and informal curricula and strengths and weaknesses of individual departments.

The development and validation of the childhood obesity prevention self-efficacy (COP-SE) survey

BACKGROUND

Physicians can play an important role in preventing and treating childhood obesity. There are currently no validated measures of medical students' self-efficacy in these skills; therefore, we sought to develop a valid and reliable computerized survey to measure medical students' self-efficacy in skills needed to prevent and treat childhood obesity.

METHODS

We developed the Childhood Obesity Prevention Self-Efficacy (COP-SE) survey with input from two expert panels and cognitive interviews with medical students. We administered the 43-item COP-SE computerized survey to a nation-wide sample of medical students.

RESULTS

The final sample consisted of 444 medical students from 53 medical schools. Exploratory factor analysis revealed a two-factor structure with a correlation of 0.637 between factors and high reliability within factors. The correlation between the COP-SE and a measure of general self-efficacy was moderate (0.648), and reliability within factors was high (Factor 1=0.946; Factor 2=0.927).

CONCLUSIONS

The 18-item COP-SE is a valid and reliable measure of childhood obesity prevention self-efficacy. Factor 1 assesses self-efficacy in nutrition counseling, and Factor 2 measures self-efficacy to assess readiness to change and initiate nutrition lifestyle changes. The correlation between the COP-SE and a measure of general self-efficacy indicates that the COP-SE is a distinct, valid assessment of domain-specific self-efficacy. The high reliability of items within factors indicates the items measure the same constructs. Therefore, medical schools can use this valid and reliable instrument as a formative or summative assessment of students' self-efficacy in childhood obesity prevention and treatment.

Testing feasibility and reliability of a set of quality indicators to evaluate the organization of palliative care across Europe: a pilot study in 25 countries

BACKGROUND

A well-organized palliative care service is a prerequisite for offering good palliative care. Reliable and feasible quality indicators are needed to monitor the quality of their organization.

AIM

To test feasibility and reliability of a previously developed set of quality indicators in settings and services that provide palliative care across Europe.

METHODS

A total of 38 quality indicators, applicable in all types of settings, rated in a RAND Delphi process, and operationalized into 38 yes/no questions, were used. Descriptives statistics, factor and reliability analyses, analysis of variance, and chi-square analyses were used.

DESIGN

Cross-sectional online survey.

SETTING/PARTICIPANTS

Questionnaires were sent to representatives of 217 palliative care settings in 25 countries. Included settings were hospices, inpatient dedicated palliative care beds, palliative care outpatient clinics, palliative care units, day care centers for palliative care, palliative care home support teams, inpatient palliative care support teams, care homes, and nursing homes.

RESULTS

All invited 25 European Association of Palliative Care countries took part. In total, 107 out of 217 participants responded (57%). The quality indicators were reduced to four coherent sub-scales, being "equipment and continuity of care," "structured documentation of essential palliative care elements in the medical record," "training and appraisal of personnel," and "availability of controlled drugs." No significant differences in quality criteria between the different types of settings and services were identified.

CONCLUSION

The set of quality indicators appeared to measure four reliable domains that assess the organization of different palliative care settings. It can be used as a starting point for quality improvement activities.

Confirmatory Factor Analysis of the System for Evaluation of Teaching Qualities (SETQ) in Graduate Medical Training

The System for Evaluation of Teaching Qualities (SETQ) was developed as a formative system for the continuous evaluation and development of physicians’ teaching performance in graduate medical training. It has been seven years since the introduction and initial exploratory psychometric analysis of the SETQ questionnaires. This study investigates the validity and reliability of the SETQ questionnaires across hospitals and medical specialties using confirmatory factor analyses (CFAs), reliability analysis, and generalizability analysis. The SETQ questionnaires were tested in a sample of 3,025 physicians and 2,848 trainees in 46 hospitals. The CFA revealed acceptable fit of the data to the previously identified five-factor model. The high internal consistency estimates suggest satisfactory reliability of the subscales. These results provide robust evidence for the validity and reliability of the SETQ questionnaires for evaluating physicians’ teaching performance.

Assessment of the clinical trainer as a role model: a Role Model Apperception Tool (RoMAT)

PURPOSE

Positive role modeling by clinical trainers is important for helping trainees learn professional and competent behavior. The authors developed and validated an instrument to assess clinical trainers as role models: the Role Model Apperception Tool (RoMAT).

METHOD

On the basis of a 2011 systematic review of the literature and through consultation with medical education experts and with clinical trainers and trainees, the authors developed 17 attributes characterizing a role model, to be assessed using a Likert scale. In 2012, general practice (GP) trainees, in their first or third year of postgraduate training, who attended a curriculum day at four institutes in different parts of the Netherlands, completed the RoMAT. The authors performed a principal component analysis on the data that were generated, and they tested the instrument's validity and reliability.

RESULTS

Of 328 potential GP trainees, 279 (85%) participated. Of these, 202 (72%) were female, and 154 (55%) were first-year trainees. The RoMAT demonstrated both content and convergent validity. Two components were extracted: "Caring Attitude" and "Effectiveness." Both components had high reliability scores (0.92 and 0.84, respectively). Less experienced trainees scored their trainers significantly higher on the Caring Attitude component.

CONCLUSIONS

The RoMAT proved to be a valid, reliable instrument for assessing clinical trainers' role-modeling behavior. Both components include an equal number of items addressing personal (Heart), teaching (Head), and clinical (Hands-on) qualities, thus demonstrating that competence in the "3Hs" is a condition for positive role modeling. Educational managers (residency directors) and trainees alike can use the RoMAT.

An argument-based approach to the validation of UHTRUST: can we measure how recent graduates can be trusted with unfamiliar tasks?

There is a need for valid methods to assess the readiness for clinical practice of medical graduates. This study evaluates the validity of Utrecht Hamburg Trainee Responsibility for Unfamiliar Situations Test (UHTRUST), an authentic simulation procedure to assess whether medical trainees are ready to be entrusted with unfamiliar clinical tasks near the highest level of Miller's pyramid. This assessment, in which candidates were judged by clinicians, nurses and standardized patients, addresses the question: can this trainee be trusted with unfamiliar clinical tasks? The aim of this paper is to provide a validity argument for this assessment procedure. We collected data from various sources during preparation and administration of a UHTRUST-assessment. In total, 60 candidates (30 from the Netherlands and 30 from Germany) participated. To provide a validity argument for the UHTRUST-assessment, we followed Kane's argument-based approach for validation. All available data were used to design a coherent and plausible argument. Considerable data was collected during the development of the assessment procedure. In addition, a generalizability study was conducted to evaluate the reliability of the scores given by assessors and to determine the proportion of variance accounted by candidates and assessors. It was found that most of Kane's validity assumptions were defendable with accurate and often parallel lines of backing. UHTRUST can be used to compare the readiness for clinical practice of medical graduates. Further exploration of the procedures for entrustment decisions is recommended.

Educational testing and validity of conclusions in the scholarship of teaching and learning

Validity and its integral evidence of reliability are fundamentals for educational and psychological measurement, and standards of educational testing. Herein, we describe these standards of educational testing, along with their subtypes including internal consistency, inter-rater reliability, and inter-rater agreement. Next, related issues of measurement error and effect size are discussed. This article concludes with a call for future authors to improve reporting of psychometrics and practical significance with educational testing in the pharmacy education literature. By increasing the scientific rigor of educational research and reporting, the overall quality and meaningfulness of SoTL will be improved.

Validity of scores from communication skills instruments for patients and their dental student-clinicians

The development of appropriate communication skills by healthcare providers is central to providing quality patient-centred care. Patients can provide valuable feedback to practitioners about their clinical communication. However, in oral health care, their involvement is uncommon and instruments specific for communication in oral health care have not been available. Recently, two complementary instruments have been developed by the Faculty of Dentistry, University of Manitoba for evaluating student-clinicians' clinical communication: one for patient evaluation and one for student self-evaluation. The aim of the current study was to provide validity evidence for the scores related to the internal structure of the revised 2007 versions of these instruments in two dental clinical/education contexts, namely the Universities of Manitoba, Canada (UM) and Adelaide, Australia (UA). The proposed factor structure and loadings, and their stability across contexts were assessed using confirmatory factor analysis, and the adequacy of the internal consistency reliability of the scores was analysed using Cronbach's alpha. The factor structure of the current 2007 versions of the patient and student instruments, derived from the previously developed longer versions of these instruments, was confirmed and was consistent across the two clinical/educational contexts. A model of partial invariance provided the best fit for these data due to variations in the magnitude of the factor loadings between sites. The internal consistency reliability of scores was high with a range of 0.88-0.97. In conclusion, the current study provides preliminary evidence regarding the validity of the scores of the current 2007 instruments, in terms of the internal structure, as measuring the five factors well. Replication of the factor structure of these instrument scores with more participants at both UA and other institutions is required.

Multisource feedback: can it meet criteria for good assessment?

INTRODUCTION

High-quality instruments are required to assess and provide feedback to practicing physicians. Multisource feedback (MSF) uses questionnaires from colleagues, coworkers, and patients to provide data. It enables feedback in areas of increasing interest to the medical profession: communication, collaboration, professionalism, and interpersonal skills. The purpose of the study was to apply the 7 assessment criteria as a framework to examine the quality of MSF instruments used to assess practicing physicians.

METHODS

The criteria for assessment (validity, reproducibility, equivalence, feasibility, educational effect, catalytic effect, and acceptability) were examined for 3 sets of instruments, drawing on published data.

RESULTS

Three MSF instruments with a sufficient body of research for inclusion-the Canadian Physician Achievement Review instruments and the United Kingdom's GMC and CFEP360 instruments-were examined. There was evidence that MSF has been assessed against all criteria except educational effects, although variably for some of the instruments. The greatest emphasis was on validity, reproducibility, and feasibility for all of the instruments. Assessments of the catalytic effect were not available for 1 of the 2 UK instruments and minimally examined for the other. Data about acceptability are implicit in the UK instruments from their endorsement by the Royal College of General Practice and explicitly examined in the Canadian instruments.

DISCUSSION

The 7 criteria provided a useful framework to assess the quality of MSF instruments and enable an approach to analyzing gaps in instrument assessment. These criteria are likely to be helpful in assessing other instruments used in medical education.