Updating conceptions of validity and reliability

Psychometric properties of the Indonesian Fonseca anamnestic index and the presence/severity of temporomandibular disorders among Indonesian young adults

OBJECTIVE

The psychometric properties of the Indonesian FAI (FAI-I) and presence/severity of temporomandibular disorders (TMDs) among Indonesian young adults were explored.

METHODS

The FAI-I was developed following the INfORM guidelines and used to determine the presence/severity of TMDs. Internal consistency/test-retest reliability were examined with Cronbach's alpha/intra-class correlation (ICC) coefficients. Construct/criteria validity were established by correlating (Spearman) the FAI-I to the five major TMD symptoms (5 Ts) and OHIP-14 (p < 0.05).

RESULTS

Five hundred-one participants (mean age 19.73 ± 1.27 years; 75.2% women) were recruited from a local University. Of these, 40.7% had no TMD, while 49.9%, 8.8%, and 0.6% had mild, moderate, and severe TMD. While Cronbach's α = 0.57 and ICC = 0.72, correlation coefficients to total 5 Ts and OHIP-14 were 0.53 and 0.47, respectively.

CONCLUSION

The FAI-I had low internal consistency, good test-retest reliability, and good validity. Moderate-to-severe TMD was experienced by 9.4% of the young adults examined.

Psychometric properties of the Maslach Burnout Inventory in healthcare professionals, Ancash Region, Peru

Background

Burnout syndrome (BS) among healthcare professionals in Peru demands immediate attention. Consequently, there is a need for a validated and standardized instrument to measure and address it effectively. This study aimed to determine the psychometric properties of the Maslach Burnout Inventory (MBI) among healthcare professionals in the Ancash region of Peru.

Methods

Using an instrumental design, this study included 303 subjects of both sexes (77.56% women), ranging in age from 22 to 68 years (M = 44.46, SD = 12.25), selected via purposive non-probability sampling. Appropriate content validity, internal structure validity, and item internal consistency were achieved through confirmatory factor analysis, and discriminant validity for the three dimensions was obtained. Evidence of convergent validity was found for the Emotional Exhaustion (EE) and Personal Accomplishment (PA) dimensions, with reliability values (ω > .75).

Results

The EE and PA dimensions exhibited acceptable levels of reliability (ω and α > .80). However, the Depersonalization (DP) dimension demonstrated significantly lower reliability (α < .60 and ω < .50).

Conclusions

A correlated three-factor model was confirmed, with most items presenting satisfactory factor loadings and inter-item correlations. Nonetheless, convergent validity was not confirmed for the DP dimension.

Academic conference posters: Describing visual impression in pharmacy education

Background

Academic conference posters are a key communication before journal articles. Attention to visual attributes can enhance academic poster communication.

Objective

This investigation's purpose was to create a visual impression measurement instrument, and then to describe and compare visual impression among scientific posters from an academic conference.

Methods

A mixed-approach rubric was created to quickly measure visual impression of academic posters. Then, posters from a pharmacy education conference were retrospectively reviewed and scored. Visual impression was compared for traditional versus contemporary poster-formats. Various poster characteristics (poster-format, summary statement presence, abstract presence, wordiness, QR-code presence, logical sequencing, visuals) that might have impacted visual communication were coded. These characteristics were regressed onto visual impression scores.

Results

Three-hundred seventy-eight posters were scored with sound inter-rater reliability. Contemporary poster-format scored significantly higher than traditional. Poster-format, abstract absence, lack of wordiness, QR-code presence, logical sequencing, and number of visuals were significant when regressed.

Conclusion

Posters at one academic conference had varied visual impression. While a contemporary poster-format appeared more helpful, it was not a panacea; variation from poor through exemplary was seen with both poster-formats. Posters are not text-filled articles; displaying a combination of visuals/text clearly and concisely can help effective communication with academic posters.

Development of a pharmacy educational environment questionnaire

INTRODUCTION

A robust instrument for measuring the educational environment in a pharmacy patient care setting is currently lacking. The authors aimed to develop a person-reported outcome measure to gauge trainee pharmacists' perceptions of their clinical workplace-based educational environment. This paper reports the various sources of validity evidence.

METHODS

A preliminary, three-domain (system, relationship, and personal), 36-item measurement instrument was developed by incorporating results from literature review, focus group interviews, iterative discussions among educators, and cognitive interviews. The preliminary instrument, together with the Dundee Ready Educational Environment Measure (DREEM) and Orientation to Happiness Scale (OHS), were administered to trainee pharmacists recruited from one pharmacy school and one healthcare cluster in Singapore. Rasch analysis was conducted on the overall instrument as well as three domains: System, Relationship, and Personal.

RESULTS

The 36-item instrument was administered to 145 learners, of which 66 were trainee pharmacists and 79 undergraduate interns. Upon removal of the middle "neutral" category to resolve poorly functioning categories and removal of five poorly functioning items, the overall instrument and its domains generally showed good fit with the Rasch model. Moreover, the overall 31-item Singapore Pharmacy Educational Environment Questionnaire (SPEEQ) and its three domains showed moderate to high convergent correlation with all the five DREEM domains and low to non-significant divergent correlation with the OHS Pleasure of Life subscale.

CONCLUSIONS

Among one sample of pharmacy trainees, the 31-item SPEEQ and its three component domains demonstrated notable validity evidence to gauge trainees' perceptions of their learning environment.

Methods to assess ambivalence towards food and diet: a scoping review

BACKGROUND

Ambivalence towards food and diet, which favours behavioural inertia, might be a barrier to adopting healthier eating behaviours. Measuring it can help researchers to better understand its relationship with behaviour change and design interventions aimed at resolving it. In this scoping review, we map and describe methods and tools employed in studies to assess, measure or classify the ambivalence of participants towards food- and diet-related attitude objects.

METHODS

In accordance with Joanna Briggs Institute guidance for conducting scoping reviews, we retrieved peer-reviewed studies from MEDLINE, CINAHL, PsycINFO, Web of Science, FSTA and Food Science Source and preprints from PsyArXiv and MedRxiv. Two independent reviewers screened the articles. We considered for inclusion peer-reviewed studies and preprints that assessed the ambivalence of participants of any age, sex or sociodemographic group towards food and diet.

RESULTS

We included 45 studies published between 1992 and 2022, which included participants from 17 countries. Eighteen methods were employed across the included studies to assess different types of ambivalence (felt, potential or cognitive-affective), the most frequent of which were the Griffin Index, the Subjective Ambivalence Questionnaire, the MouseTracker Paradigm and the Orientation to Chocolate Questionnaire.

CONCLUSION

This scoping review identified several methods and tools to assess different types of ambivalence towards food- and diet-related objects, providing an array of options for future studies.

Using Rasch measurement for instrument rating scale refinement

OUR SITUATION

Rasch measurement is an analysis tool that can provide validity evidence for instruments that attempt to measure student learning or other psychosocial behaviors, regardless if tools are newly created, modified, or previously developed. Rating scales are exceedingly common among psychosocial instruments and properly functioning rating scales are critical to effective measurement. Rasch measurement can help investigate this.

METHODOLOGICAL LITERATURE REVIEW

Aside from using Rasch measurement from the beginning to help create rigorous new measurement instruments, researchers can also benefit from employing Rasch measurement on previously developed instruments that had not included Rasch measurement during development. This article is focused on Rasch measurement's unique analysis of rating scales. That is, Rasch measurement can uniquely help examine if and how an instrument's rating scale is functioning among newly studied respondents (who will likely differ from the originally researched sample).

OUR RECOMMENDATIONS AND THEIR APPLICATION

After reviewing this article, the reader should be able to describe Rasch measurement, including how it is focused on fundamental measurement and how it differs from classical test theory and item-response theory, and reflect on situations in their own research where a Rasch measurement analysis might be helpful for generating validation evidence with a previously developed instrument.

POTENTIAL IMPACT

In the end, Rasch measurement can offer a helpful, unique, rigorous approach to further developing instruments that scientifically measure, accurately and precisely.

Development and content validity of a rating scale for the pain and disability drivers management model

Background

Establishing the biopsychosocial profile of patients with low back pain (LBP) is essential to personalized care. The Pain and Disability Drivers Management model (PDDM) has been suggested as a useful framework to help clinicians establish this biopsychosocial profile. Yet, there is no tool to facilitate its integration into clinical practice. Thus, the aim of this study is to develop a rating scale and validate its content, to rapidly establish the patient’s biopsychosocial profile, based on the five domains of the PDDM.

Methods

The tool was developed in accordance with the principles of the COSMIN methodology. We conducted three steps: 1) item generation from a comprehensive review, 2) refinement of the scale with clinicians’ feedback, and 3) statistical analyses to assess content validity.

To validate the item assessing with Likert scales, we performed Item level-Content Validity Index (I-CVI) analyses on three criteria (clarity, presentation and clinical applicability) with an a priori threshold of > 0.78. We conducted Average-Content Validity Index (Ave-CVI) analyses to validate the overall scale with a threshold of > 0.9.

Results

In accordance with the PDDM, we developed a 5-item rating scale (1 per domain) with 4 score options. We selected clinical instruments to screen for the presence or absence of problematic issues within each category of the 5 domains. Forty-two participants provided feedback to refine the scale’s clarity, presentation, and clinical applicability. The statistical analysis of the latest version presented I-CVI above the threshold for each item (I-CVI ranged between 0.94 and 1). Analysis of the overall scale supported its validation (Ave-CVI = 0.96 [0.93;0.98]).

Conclusion

From the 51 biopsychosocial elements contained within the 5 domains of the PDDM, we developed a rating scale that allows to rapidly screen for problematic issues within each category of the PDDM’s 5 domains. Involving clinicians in the process allowed us to validate the content of the first scale to establish the patient’s biopsychosocial profile for people with low back pain. Future steps will be necessary to continue the psychometric properties analysis of this rating scale.

Supplementary Information

The online version contains supplementary material available at 10.1186/s40945-022-00137-2.

A Situational Judgment Test to Assess Students' Achievement of Accreditation Council for Pharmacy Education Standards 3 and 4

Objective. For pharmacy students to successfully meet competencies related to Accreditation Council for Pharmacy Education Standards 3 and 4, it is essential for pharmacy programs to assess student progression in the affective domain. The purpose of this study was to develop and assess the validity of a situational judgment test (SJT) for evaluating student mastery of Standards 3 and 4.Methods. A multi-institutional faculty team developed an 18-item SJT that consisted of scenarios asking the respondent to rank the effectiveness of four response options mapped to Standards 3 and 4. The research team systematically reviewed the literature, created items, and deliberated until consensus was achieved. Subject matter experts (SMEs) reviewed and provided feedback on the instrument. Students from two institutions were recruited to participate in cognitive interviews about the finalized instrument. Cognitive interview data were analyzed to identify themes.Results. After edits were made to the instrument based on SME feedback, students (n=18) in the cognitive interviews identified item length as a concern and commented on item/response clarity and comprehensiveness. Data from the cognitive interviews were used to modify the SJT to reduce the length and clarify items. The result was two shorter versions of the instrument, both with similar mapping to all elements in Standards 3 and 4.Conclusion. Early steps in validating the SJT suggested that the instrument may be a promising tool to assess student progression in the affective domain. The SJT instrument is intended to provide evidence of student pharmacist development that occurs in the didactic, experiential, and co-curricular portions of pharmacy education. The instrument can serve as one part of a comprehensive assessment plan.

A Teachable Moment Matters for: Striving for inter-rater reliability with a patient counseling assessment rubric: Lessons learned

The Teachable Moments Matter (TMM) category of articles is designed to offer readers insight into a methodological issue identified within a companion article. Written in collaboration with one of these authors, these articles provide an opportunity to focus on a challenge experienced by the authors and, in the process, provide one or more perspectives as to how to successfully navigate this issue. The current TMM is focused on issues and pitfalls in validation. The Journal hopes this case-based approach will help highlight the nuance of a topic in context, something that might get "lost" in the entirety of a full-length article.

Psychometric evaluation of the PROMIS social function short forms in Chinese patients with breast cancer

Background

The diagnosis of breast cancer and the subsequent treatment undermine patients’ participation in social activities. This study aimed to carry out a cross-cultural adaption and analysis of the construct validity and reliability of the Chinese version of the PROMIS social function short forms in patients with breast cancer.

Methods

This study utilized a cross-sectional research design, and was registered in the Chinese Clinical Trial Registry (ChiCTR2000035439). After a standardized cross-cultural adaption process, a psychometric evaluation was performed of the Chinese version of the PROMIS social function short forms. Using convenience sampling, eligible patients with breast cancer from tertiary hospitals in China were enrolled from January 2019 to July 2020. Participants completed the sociodemographic information questionnaire, the PROMIS social function short forms, the Functional Assessment of Cancer Therapy-Breast, the PROMIS emotional support short form and the PROMIS anxiety short form.

Results

Data were collected from a sample of 633 patients whose mean age was 48.1 years. The measures showed an absence of floor and ceiling effects. Regarding construct validity, the results of confirmatory factor analysis supported the original two-factor structure of the PROMIS social function short forms. In addition, the measures were found to have acceptable known-group validity, measurement invariance, and convergent and discriminate validity. Regarding reliability, the Cronbach’s α was high for all items (> 0.70).

Conclusion

The Chinese version of the PROMIS social function short forms was demonstrated to be a valid and reliable measure for the assessment of social function in Chinese patients with breast cancer. Additional psychometric evaluation is needed to draw firm conclusions.

Validity based on content: A challenge in health measurement scales

Content-based validity can be defined as the evidence of the degree to which the items of an instrument represent the construct to be inferred. It is the initial and unavoidable step for evaluating other sources of validity. However, it is one of the least reported psychometric properties in development of health measurement scales. That is why it is considered a challenge that must be overcome with the recognition of its value. In this narrative review, the concept of content as evidence of validity is revised and methods for evaluation are synthesized with an algorithm.