Measurement versus prediction in the construction of patient-reported outcome questionnaires: can we have our cake and eat it?

Leading indicators of readiness among the general Army and Special Operations Forces: Predictive and psychometric analysis of the Global Assessment Tool

Self-assessments are commonly used to track Army readiness in specialized communities, but they are rarely analyzed for reliability and predictive validity. Before introducing new assessments, existing ones should be reevaluated. We examined the Global Assessment Tool (GAT), an annual Army-required self-assessment with multiple psychosocial and health behavior short scales. Psychometric analyses on nine scales included item response theory (IRT) and measurement invariance models across total Army (n = 743,057) and special operations forces (SOF; n = 3,478) cohorts. Predictive analyses examined demographic-adjusted associations between GAT scales and one-year incident medical non-readiness (MNR). Most scales had adequate reliability, although some exhibited highly skewed distributions, which likely increased measurement error. Most scales exhibited metric and scalar measurement equivalence across total Army and SOF groups. Scores from scales measuring positive characteristics were associated with lower odds of MNR (good coping, flexibility, optimism, positive affect, work engagement, friendship, organization trust; adjusted odds ratios ≤ 0.75); scores from scales measuring negative characteristics were associated with increased odds of MNR (poor sleep, depression, negative affect, loneliness; adjusted odds ratios ≥ 1.4). Associations were similar across Army and SOF cohorts. In conclusion, self-report data can potentially contribute to command surveillance, but iterative quality-checks are necessary after deployment.

A systematic review of measurement uncertainty visualizations in the context of standardized assessments

This systematic review summarized findings of 29 studies evaluating visual presentation formats appropriate for communicating measurement uncertainty associated with standardized clinical assessment instruments. Studies were identified through systematic searches of multiple databases (Medline, Embase, PsycInfo, ERIC, Scopus, and Web of Science). Strikingly, we found no studies which were conducted using samples of clinicians and included clinical decision-making scenarios. Included studies did however find that providing participants with information about measurement uncertainty may increase awareness of uncertainty and promote more optimal decision making. Formats which visualize the shape of the underlying probability distribution were found to promote more accurate probability estimation and appropriate interpretations of the underlying probability distribution shape. However, participants in the included studies did not seem to benefit from the additional information provided by such plots during decision-making tasks. Further explorations into how presentations of measurement uncertainty impact clinical decision making are needed to examine whether findings of the included studies generalize to clinician populations. This review provides an important overview of pitfalls associated with formats commonly used to communicate measurement uncertainty in clinical assessment instruments, and a potential starting point for further explorations into promising alternatives. Finally, our review offers specific recommendations on how remaining research questions might be addressed.

Smart classroom learning environment preferences of higher education teachers and students in China: An ecological perspective

By evaluating learners' perceived preferences for the learning environment, we can understand the important characteristics and better improve the learning environment, ultimately to provide great potential for the optimization of teaching practice. Seeing that the current research pays less attention to teachers' and students' preferences for the space environment simultaneously, based on the survey of 1937 undergraduates and 107 teachers from a university in central China, this study aims to explore their preferences for smart learning environment. Based on the ecological theory and research results of the existing learning environment, this paper constructed an ecological model and a conceptual model of learning space preferences. An empirical study was conducted to explore the impact of sociodemographic variables on personal spatial preference. The results showed that teachers and students had a positive attitude towards the smart learning environment, and gender, age, grade, subject category and other variables had limited impact on spatial preference.

Dissatisfaction With Total Knee Arthroplasty at 1 Year Post Surgery Can be Predicted Using a Short Questionnaire Early in the Recovery Process

BACKGROUND

Approximately 20% of patients are dissatisfied with their total knee arthroplasty (TKA). Identifying patients likely to be dissatisfied early in the recovery process could help reduce the number of dissatisfied patients. The purpose of this study was to create an easily administered short questionnaire to identify patients likely to be dissatisfied at 1 year post surgery early in the recovery process.

METHODS

The study included 275 patients who underwent primary TKA for osteoarthritis. Individual 3-month postsurgery questionnaire items from the Knee Injury and Osteoarthritis Outcome Score and Knee Society Knee Scoring System were pooled together and used as candidate items to create 3 different short questionnaires. Items included in each questionnaire were selected using least absolute shrinkage and selection operator logistic regressions, a backward elimination method, and theory-based approaches. The area under the curve for each short questionnaire was calculated to evaluate predictive performances.

RESULTS

All 3 questionnaires contained a small number of items and appeared to successfully predict 1-year postsurgery dissatisfaction early in the recovery process. The least absolute shrinkage and selection operator logistic regression, backward elimination, and theory-based questionnaires were comprised of 4, 7, and 5 items and had the area under the curve scores of 0.893, 0.902, and 0.890, respectively. A question evaluating rising from sitting and activities of daily living appeared in all of the created questionnaires.

CONCLUSION

A short questionnaire that is easy to administer and interpret can effectively predict TKA patient dissatisfaction at 1 year post surgery early in the recovery process.

Using validity theory and psychometrics to evaluate and support expanded uses of existing scales

BACKGROUND

Scale development is a complex activity requiring significant investments of time and money to produce evidence of a scale's ability to produce reliable scores and valid inferences. With increasing use of clinical outcome assessments (COAs) in medical product development, evidentiary expectations of regulatory bodies to support inferences are a key consideration. The goal of this paper is to demonstrate how existing methods in measurement science can be used to identify and fill evidence gaps when considering re-purposing an existing scale for a new use case (e.g., new patient population, altering the recall period), rather than creating a new COA tool.

METHODS

We briefly review select validity theory and psychometric concepts, linking them to the nomenclature in the COA/regulated space. Four examples (two in-text and two in online supplemental materials) of modifications are presented to demonstrate these ideas in practice for quality of life (QOL)-related measures.

RESULTS

Each example highlights the initial process of evaluating the desired validity claims, identifying gaps in evidence to support these claims, and determining how such gaps could be filled, often without having to develop a new measure.

CONCLUSIONS

If an existing scale, with minimal modification or additional evidence, can be shown to be fit for a new purpose, considerable effort can be saved and research waste avoided. In many cases, a new instrument is simply unnecessary. Far better to recycle an "old" scale for a new use-with sufficient evidence that it is fit for that purpose-than to "buy" a new one.

Academic expectations among international students from North-Western China: A case of technology use during and post COVID-19

This study examines the influence of the COVID-19 crisis on academic expectations among international students from north-western China. According to past studies, academic expectations are multifaceted, making it critical to test the methods employed to assess this fundamental trait. The outbreak of COVID-19 has resulted in various significant changes in education, which have shifted from traditional to online or mixed formats. As a result, examining international students' academic expectations along with their interactions with adopted technologies is a topic that addresses the current situation and issues. A mixed approach, comprising two different instruments (questionnaire and interview), was followed to achieve this primary objective. While a survey with a questionnaire was undertaken with 551 international students, divided into two groups, ten students were interviewed during and after the lockdowns. The findings revealed that COVID-19 had a significant impact on the academic expectations of students as well as many elements such as training for employment, personal and social development, international student mobility, motivation, social pressure, and social interaction with the help of supporting technologies. In terms of gender, men outperformed women in motivation, social interaction, training for employment, and personal and social development factors. Similarly, as per the grade variable (undergraduate, postgraduate, doctoral), the same higher trend was seen in postgraduates. Based on these findings, a set of recommendations was put forward. In the future, technology will be helpful in China's educational sector, such as online group collaboration, open education, managing student retention, and supervising teachers' recruitment.

Negative effects during multicomponent group-based treatment: A multisite study

Objective Negative effects (NEs) in group treatments remain an under-researched area. This study aimed to explore the prevalence of various types of NEs in a multicomponent group-based treatment and to determine their predictors. Method: A total of 330 patients participating in a multicomponent group-based treatment were recruited across seven clinical sites. At the end of treatment, the Negative Effects Questionnaire (NEQ) was used to measure NEs. Item-level descriptive analysis was conducted to explore the prevalence of various types of NEs, and structural equation modeling was used to determine predictors of these NEs. Results: The most frequently reported type of NEs was the worsening of symptoms, and the single most frequently reported item was the resurfacing of unpleasant memories. Predictors of NEs included the overall distress level, alexithymia, attachment avoidance, low working alliance, problem actuation, and worse outcomes; psychological mindedness was a protective factor. Conclusion: Patients who experience higher levels of distress at the beginning of treatment, who perceive the group working alliance as problematic, and who experience high in-session emotional arousal related to their problem seem to be especially prone to reporting NEs. Furthermore, the findings do not support the assumption that NEs are a prerequisite for therapeutic change.Trial registration: ISRCTN.org identifier: ISRCTN13532466.

The prevalence of the academic learning difficulties: An observation tool

This study seeks to develop an effective observation tool to determine the prevalence of various academic learning difficulties among school students at the primary level in Sana'a City, Yemen. A measure comprising of 34 items has been processed by the EFA and CFA for contriving ALD's psychometric properties. The study sample comprised 714 students between 6 - 14 years of age. The study's findings revealed that the observation tool under development could measure the prevalence of various academic learning difficulties to a great extent with accuracy. The learning difficulties were classified under five categories based on observation scores. The observed raw scores were standardised after taking the standard deviation from the sample's mean value into consideration. The study's findings suggested that the gender and grade of the subjects affected academic learning difficulties significantly. A brief discussion of the educational implications of these findings has also been presented.

Predicting the Development of Adult Nature Connection Through Nature Activities: Developing the Evaluating Nature Activities for Connection Tool

Nature Connection (NC) is considered an important driver of conservation behavior. Consequently, conservation organizations run many activities aiming to increase NC among participants. However, little is known about which activities are most effective at doing this and why. This study developed the Evaluating Nature Activities for Connection Tool (ENACT), to evaluate the effectiveness of activities for increasing participants' NC and nature-related intentions. ENACT comprises 11 activity aspects identified through two research phases. In Phase 1, a literature search, focus group and interviews identified desired, short-term behavioral outcomes of nature activities, and variables that might promote these. In Phase 2, 241 adults completed a pilot survey immediately post-nature activity, with 1-month follow-up (N = 145), to evaluate the impact of participation on NC, nature-related behavioral intentions and behaviors. ENACT correlated with NC measures and offered incremental validity in predicting nature-related behavioral intentions and self-reported behaviors after 1 month.

Psychometric evaluation of a patient-reported item bank for healthcare engagement

PURPOSE

Healthcare engagement is a core measurement target for efforts to improve healthcare systems. This construct is broadly defined as the extent to which healthcare services represent collaborative partnerships with patients. Previous qualitative work operationalized healthcare engagement as generalized self-efficacy in four related subdomains: self-management, collaborative communication, health information use, and healthcare navigation. Building on this work, our objective was to establish a healthcare engagement instrument that is sufficiently unidimensional to yield a single score.

METHOD

We conducted cognitive interviews followed by a nation-wide mail survey of US Veteran Administration (VA) healthcare users. Data were collected on 49 candidate healthcare engagement items, as well as measures of self-efficacy for managing symptoms, provider communication, and perceived access. Items were subjected to exploratory bifactor, statistical learning, and IRT analyses.

RESULTS

Cognitive interviews were completed by 56 patients and 9552 VA healthcare users with chronic conditions completed the mail survey. Participants were mostly white and male but with sizable minority participation. Psychometric analyses and content considerations reduced the item pool to 23 items, which demonstrated a strong general factor (OmegaH of .89). IRT analyses revealed a high level of reliability across the trait range and little DIF across groups. Most health information use items were removed during analyses, suggesting a more independent role for this domain.

CONCLUSION

We provide quantitative evidence for a relatively unidimensional measure of healthcare engagement. Despite developed with VA healthcare users, the measure is intended for general use. Future work includes short-form development and validation with other patient groups.

Prediction of child and adolescent outcomes with broadband and narrowband dimensions of internalizing and externalizing behavior using the child and adolescent version of the Strengths and Difficulties Questionnaire

The Strengths and Difficulties Questionnaire (SDQ) is a frequently used screening instrument for behavioral problems in children and adolescents. There is an ongoing controversy—not only in educational research—regarding the factor structure of the SDQ. Research results speak for a 3-factor as well as a 5-factor structure. The narrowband scales (5-factor structure) can be combined into broadband scales (3-factor structure). The question remains: Which factors (narrowband vs. broadband) are better predictors? With the prediction of child and adolescent outcomes (academic grades, well-being, and self-belief), we evaluated whether the broadband scales of internalizing and externalizing behavior (3-factor structure) or narrowband scales of behavior (5-factor structure) are better suited for predictive purposes in a cross-sectional study setting. The sample includes students in grades 5 to 9 (N = 4642) from the representative German Health Interview and Examination Survey for Children and Adolescents (KiGGS study). The results of model comparisons (broadband scale vs. narrowband scales) did not support the superiority of the broadband scales with regard to the prediction of child and adolescent outcomes. There is no benefit from subsuming narrowband scales (5-factor structure) into broadband scales (3-factor structure). The application of narrowband scales, providing a more differentiated picture of students’ academic and social situation, was more appropriate for predictive purposes. For the purpose of identifying students at risk of struggling in educational contexts, using the set of narrowband dimensions of behavior seems to be more suitable.

Evaluating the Measurement Invariance of MMPI-2-RF Restructured Clinical Scale 4 (Antisocial Behavior) between American and Korean Clinical Samples: Exploring Cultural and Translation Issues Affecting Item Responding

Establishing the cross-cultural measurement invariance of psychometric scales is considered an essential step before scale means are compared across cultures. Although the MMPI instruments have been extensively researched, few studies have examined the measurement equivalence of MMPI scales in cross-cultural research. This study examined the measurement invariance of MMPI-2-RF Restructured Clinical Scale 4 (RC4; Antisocial Behavior) using multi-group confirmatory factor analysis with American and Korean clinical samples by (a) comparing a rationally-derived four-factor model (School Problems, Substance Abuse, Family Problems, and Violation of Social Norms) with a one-factor model, and (b) examining the measurement invariance of the RC4 four-factor model. After adjusting for age and gender, partial scalar invariance was achieved, and six non-invariant items were identified, most of which centered around substance abuse. Results support the generalizability of the four factors across cultures; however, special attention is needed when using substance abuse items with Korean clinical populations. Plausible sources of item non-invariance were explored in the context of translation challenges and observed patterns of relationship with external measures.

Psychometric Properties of the Externalizing Spectrum Inventory: Replication and Extension across Clinical and Non-Clinical Samples

The Externalizing Spectrum Inventory aims at assessing personality features that underlie externalizing disorders such as substance abuse and antisocial behaviors. The objective was to replicate the psychometric properties of the 160-item Externalizing Spectrum Inventory in Dutch clinical and non-clinical samples. First, Cronbach's alpha, test-retest reliability and the factor structure were analyzed on a mixed sample of inpatients (n = 149), undergraduates (n = 227), and community participants (n = 178). The factor structure was evaluated through confirmatory and exploratory factor analyses; for the latter Parallel Analysis was used, based on Minimum Rank Factor Analysis. Next, the criterion validity was analyzed using the Aggression Questionnaire and the NEO-Five Factor Inventory as external measures. The Dutch Externalizing Spectrum Inventory subscales showed sufficient reliability (α=.68-.94; ICC=.68-.91), except in the undergraduate sample (α=.49-.96; ICC=.43-.97). The factor structure of the Externalizing Spectrum Inventory was not confirmed and the exploratory analysis yielded different factor solutions across samples. The criterion validity was supported with regard to trait aggression and partly supported with regard to the Five Factor Model. The results suggest that the ESI-160 and its original factor model can be used for prediction purposes. However, further research of the factor structure is strongly recommended.

A Comparison of Short Forms of the Screener and Opioid Assessment for Patients With Pain – Revised (SOAPP-R)

Abstract. The Screener and Opioid Assessment for Patients with Pain – Revised (SOAPP-R) is a 24-item self-report questionnaire that assesses risk of aberrant medication-related behavior among chronic pain patients. Recently, an 8-item version of the SOAPP-R that weights items differentially was proposed. However, no previous study had compared the 8-item form with other short versions of the SOAPP-R, including a static 12-item short form and computer-based versions customizing the test length to the individual respondent. Moreover, no prior research had investigated combining the 8-item short form with customized computer-based stopping rules to further enhance efficiency. The objectives of this study were to compare the 8-item version with previously recommended short forms of the SOAPP-R, and to develop and evaluate a new version of the SOAPP-R combining the 8-item version with computer-based stopping rules. Versions were compared via sensitivity, specificity, and mean test length using real-data simulation of three datasets. Although results varied across datasets, the 8-item SOAPP-R compared favorably to previously recommended forms. Combining the 8-item form with computer-based stopping rules reduced the mean test length without affecting sensitivity or specificity; thus, the combined approach is recommended. The methodology used to shorten questionnaires via computer-based testing can also be applied to other instruments.

Methods for questionnaire design: a taxonomy linking procedures to test goals

BACKGROUND

In the clinical field, the use of questionnaires is ubiquitous, and many different methods for constructing them are available. The reason for using a specific method is usually lacking, and a generally accepted classification of methods is not yet available. To guide test developers and users, this article presents a taxonomy for methods of questionnaire design which links the methods to the goal of a test.

METHODS

The taxonomy assumes that construction methods are directed towards psychometric aspects. Four stages of test construction are distinguished to describe methods: concept analysis, item production, scale construction, and evaluation; the scale construction stage is used for identifying methods. It distinguishes six different methods: the rational method utilizes expert judgments to ensure face validity. The prototypical method uses prototypicality judgments to ensure process validity. In the internal method, item sets are selected that optimize homogeneity. The external method optimizes criterion validity by selecting items that best predict an external criterion. Under the construct method theoretical considerations are used to optimize construct validity. The facet method is aimed at optimizing content validity through a complete representation of the concept domain.

CONCLUSION

The taxonomy is comprehensive, constitutes a useful tool for describing procedures used in questionnaire design, and allows for setting up a test construction plan in which the priorities among psychometric aspects are made explicit.

Alcohol screening and assessment measures for young people: A systematic review and meta-analysis of validation studies

BACKGROUND

There is a strong rationale for clinicians to identify risky drinking among young people given the harms caused by alcohol. This systematic review evaluates the quality of evidence in the validation literature on alcohol screening and assessment measures for young people under 25.

METHODS

Six electronic databases (MEDLINE; EMBASE; PsycINFO; SSCI; HMIC; ADAI) were searched in May 2016 for published and grey literature. Full-text reports published in English since 1980 were included if they aimed to validate an alcohol screening or assessment measure in comparison with a previously validated alcohol measure. Risk of bias was assessed in studies surpassing a priori quality thresholds for predictive validity, internal and test-retest reliability using COSMIN and QUADAS-2.

RESULTS

Thirty nine reports comprising 135 discrete validation studies were included. Summary estimates indicated that the screening instruments performed well - AUC 0.91 (95% CI: 0.88 to 0.93); sensitivity 0.98 (0.95 to 0.99); specificity 0.78 (0.74 to 0.82). Noting a paucity of validation evidence for existing assessment instruments, aggregated reliability estimates suggest a reliability of 0.81 (0.78 to 0.83) adjusted for 10 items. Risk of bias was high for both types of studies.

CONCLUSIONS

The volume and quality of available evidence are superior for screening measures. It is recommended that clinicians use alcohol frequency or quantity items if asking a single question. If there is an opportunity to ask more questions either the 3-item AUDIT-C or the 10-item AUDIT are recommended. There is a need to develop new instruments to assess young people's alcohol-related problems.

Emotional and Behavioral Problems in Children and Adolescents in Central Kenya

Emotional and behavioral problems (EBP) during childhood and adolescence are a common concern for parents and mental health stakeholders. However, little has been documented about their prevalence in Kenyan children and adolescents. This study aimed to close this gap. The study included Child Behavior Checklist reports from 1022 Kenyan parents on their children (ages 6-18 years) and Youth Self-Reports from 533 adolescents (ages 12-18) living in Kenya's Central Province. EBP in Kenya are highly prevalent compared to multi-cultural standards for parent reports, with 27 and 17% scoring in the borderline and clinical range, respectively. Based on parent reports, younger children scored higher on EBP than older children, and higher on internalizing problems. Based on self-reports girls scored higher than boys, particularly on internalizing problems. The study provides evidence on elevated parent-reported EBP in Kenyan youths. Mental health providers should focus on interventions that reduce EBP in Kenyan youths.

Some recommendations for developing multidimensional computerized adaptive tests for patient-reported outcomes

PURPOSE

Multidimensional item response theory and computerized adaptive testing (CAT) are increasingly used in mental health, quality of life (QoL), and patient-reported outcome measurement. Although multidimensional assessment techniques hold promises, they are more challenging in their application than unidimensional ones. The authors comment on minimal standards when developing multidimensional CATs.

METHODS

Prompted by pioneering papers published in QLR, the authors reflect on existing guidance and discussions from different psychometric communities, including guidelines developed for unidimensional CATs in the PROMIS project.

RESULTS

The commentary focuses on two key topics: (1) the design, evaluation, and calibration of multidimensional item banks and (2) how to study the efficiency and precision of a multidimensional item bank. The authors suggest that the development of a carefully designed and calibrated item bank encompasses a construction phase and a psychometric phase. With respect to efficiency and precision, item banks should be large enough to provide adequate precision over the full range of the latent constructs. Therefore CAT performance should be studied as a function of the latent constructs and with reference to relevant benchmarks. Solutions are also suggested for simulation studies using real data, which often result in too optimistic evaluations of an item bank's efficiency and precision.

DISCUSSION

Multidimensional CAT applications are promising but complex statistical assessment tools which necessitate detailed theoretical frameworks and methodological scrutiny when testing their appropriateness for practical applications. The authors advise researchers to evaluate item banks with a broad set of methods, describe their choices in detail, and substantiate their approach for validation.