The potential synergy between cognitive models and modern psychometric models

Semantic primes theory may be helpful in designing questionnaires such as to prevent response shift

OBJECTIVES

The purpose of randomized control trials (RCTs) can be the assessment of the direct effect of treatment on health-related quality of life (HRQL). Response shift (RS) theory considers that a change in HRQL scores observed over time cannot be explained solely by a direct effect of a medical condition, it may also result from a change in the way people appraise their HRQL. The RS effect is a potential bias that is liable to compromise efficient assessment of the effect of treatment on HRQL.

STUDY DESIGN AND SETTING

We hypothesize a link between the RS effect on HRQL scores and the level of complexity of HRQL conceptualization.

RESULTS

We discuss how the impact of reconceptualization on scores depends on the complexity of the linguistic definition of a subjective construct and how for reprioritization the impact depends on the dimensionality. The linguistic theory of semantic primes is used to help identify how subjective constructs can be classified according to the complexity of their definitions.

CONCLUSION

Finally, we suggest that the impact of the RS effect on HRQL scores could be avoided (or lessened) if questionnaires were designed with a rule of "the least semantic and psychometric complexity" in mind.

The quality of life of people in chronic pain: developing a pain and discomfort module for use with the WHOQOL

This article reports the development of a pain and discomfort module (PDM) designed to assess the full impact of quality of life (QoL) relating to chronic pain, which could be used with the generic World Health Organisation Quality of Life Assessment (WHOQOL). First, cognitive interviewing was completed with nine participants with chronic pain, for 108 items representing 10 pain-specific facets of QoL. Sixty-eight QoL items and 16 importance questions on pain were relevant, comprehensive, comprehensible and acceptable to users, and were confirmed to assess their purported concepts. Secondly, these items were pilot tested using a cross-sectional survey of 216 people with chronic pain, to investigate the preliminary psychometric properties of the PDM, and reduce its items statistically. All new facets were important to those with chronic pain. Sixteen items within four facets of pain relief, anger and frustration, vulnerability/fear/worry, and uncertainty were retained, and demonstrated acceptable to good internal consistency reliability (α = 0.77-0.85). The PDM is a self-administered, multidimensional subjective assessment of pain-related QoL, with potential to evaluate pain-relieving interventions, identify sufferer's needs, and for survey use.

Self-reported health of people with intellectual disability

Self-reported health is an important outcome in the evaluation of health care but is largely ignored in favor of proxy-based reporting for people with an intellectual disability. This study briefly reviews the role of self-report in health assessment of people with intellectual disability and the challenges and recommendations that have emerged from the considerable body of research on interviewing and self-report. Limitations in current recommendations are addressed from the perspective of the cognition of self-report. The review describes conceptual directions for the reconciliation of the two contradictory themes in the treatment of self-report: the centrality given to personal perceptions and choices and the methodological concerns over the meaningfulness and validity of the self-reporting process.

The differential item functioning and structural equivalence of a nonverbal cognitive ability test for five language groups

Orientation: For a number of years, eliminating a language component in testing by using nonverbal cognitive tests has been proposed as a possible solution to the effect of groups’ languages (mother tongues or first languages) on test performance. This is particularly relevant in South Africa with its 11 official languages.Research purpose: The aim of the study was to determine the differential item functioning (DIF) and structural equivalence of a nonverbal cognitive ability test (the PiB/SpEEx Observance test [401]) for five South African language groups.Motivation for study: Cultural and language group sensitive tests can lead to unfair discrimination and is a contentious workplace issue in South Africa today. Misconceptions about psychometric testing in industry can cause tests to lose credibility if industries do not use a scientifically sound test-by-test evaluation approach.Research design, approach and method: The researcher used a quasi-experimental design and factor analytic and logistic regression techniques to meet the research aims. The study used a convenience sample drawn from industry and an educational institution.Main findings: The main findings of the study show structural equivalence of the test at a holistic level and nonsignificant DIF effect sizes for most of the comparisons that the researcher made.Practical/managerial implications: This research shows that the PIB/SpEEx Observance Test (401) is not completely language insensitive. One should see it rather as a language-reduced test when people from different language groups need testing.Contribution/value-add: The findings provide supporting evidence that nonverbal cognitive tests are plausible alternatives to verbal tests when one compares people from different language groups.

A randomized trial of mailed questionnaires versus telephone interviews: response patterns in a survey

Background

Data for health surveys are often collected using either mailed questionnaires, telephone interviews or a combination. Mode of data collection can affect the propensity to refuse to respond and result in different patterns of responses. The objective of this paper is to examine and quantify effects of mode of data collection in health surveys.

Methods

A stratified sample of 4,000 adults residing in Denmark was randomised to mailed questionnaires or computer-assisted telephone interviews. 45 health-related items were analyzed; four concerning behaviour and 41 concerning self assessment. Odds ratios for more positive answers and more frequent use of extreme response categories (both positive and negative) among telephone respondents compared to questionnaire respondents were estimated. Tests were Bonferroni corrected.

Results

For the four health behaviour items there were no significant differences in the response patterns. For 32 of the 41 health self assessment items the response pattern was statistically significantly different and extreme response categories were used more frequently among telephone respondents (Median estimated odds ratio: 1.67). For a majority of these mode sensitive items (26/32), a more positive reporting was observed among telephone respondents (Median estimated odds ratio: 1.73). The overall response rate was similar among persons randomly assigned to questionnaires (58.1%) and to telephone interviews (56.2%). A differential nonresponse bias for age and gender was observed. The rate of missing responses was higher for questionnaires (0.73 – 6.00%) than for telephone interviews (0 – 0.51%). The "don't know" option was used more often by mail respondents (10 – 24%) than by telephone respondents (2 – 4%).

Conclusion

The mode of data collection affects the reporting of self assessed health items substantially. In epidemiological studies, the method effect may be as large as the effects under investigation. Caution is needed when comparing prevalences across surveys or when studying time trends.

Validity of patient-reported health-related quality of life global ratings of change using structural equation modeling

BACKGROUND

Patient-perceived global ratings of change are often used as anchors of health-related quality of life (HRQoL) since they are easy for clinicians to interpret and incorporate the patient's perception of change as a means to capture clinical significance. Although this approach may be preferred, the validity of the anchor-based approach is currently under scrutiny.

OBJECTIVE

To estimate the explained variation in single-item domain-specific global ratings of change (GRCs) that is accounted for by time 1 (T1) and time 2 (T2) domain-specific summary change scores from the Short-Form 36, V2 (SF-36) Health Survey in asthma primary care patients.

METHODS

The baseline and first follow-up enrollment data to be evaluated in this investigation were part of a larger longitudinal HRQoL study conducted from August 2000-December 2002, in which the 356 asthma patients from Midwestern primary care facilities completed telephone interviews for every two consecutive months for a year on multiple HRQoL measures, including the SF-36 and domain-specific GRCs. A structural equation modeling technique was employed to ascertain the explained variability in patient-reported GRCs for each SF-36 domain that is accounted for by the summary change scores at the two time-points for four SF-36 domains (bodily pain, general health perception, mental health, and physical functioning). The model was estimated by the maximum likelihood method with the Satorra-Bentler correction for ordinal variables using equal threshold asymptotic covariance matrices.

RESULTS

Multicollinearity between T1 and T2 latent constructs clouded interpretation of the standardized structural coefficients leading to GRCs. Correlations, however, revealed that all four domain-specific GRCs were more strongly related to T2- than T1-domain summary scores, indicating that patients were not equally relying on T1 and T2 to generate the GRCs. Furthermore, T1-domain summary scores were not of equal magnitude and opposite sign as compared to T2 scores.

CONCLUSIONS

In this study, there is insufficient evidence to establish SF-36 domain-specific GRC validity in asthma primary care patients. Therefore, it is recommended to reassess validity before using domain-specific SF-36 GRCs to classify clinically important change over time.

Developing tailored instruments: item banking and computerized adaptive assessment

Item banks and Computerized Adaptive Testing (CAT) have the potential to greatly improve the assessment of health outcomes. This review describes the unique features of item banks and CAT and discusses how to develop item banks. In CAT, a computer selects the items from an item bank that are most relevant for and informative about the particular respondent; thus optimizing test relevance and precision. Item response theory (IRT) provides the foundation for selecting the items that are most informative for the particular respondent and for scoring responses on a common metric. The development of an item bank is a multi-stage process that requires a clear definition of the construct to be measured, good items, a careful psychometric analysis of the items, and a clear specification of the final CAT. The psychometric analysis needs to evaluate the assumptions of the IRT model such as unidimensionality and local independence; that the items function the same way in different subgroups of the population; and that there is an adequate fit between the data and the chosen item response models. Also, interpretation guidelines need to be established to help the clinical application of the assessment. Although medical research can draw upon expertise from educational testing in the development of item banks and CAT, the medical field also encounters unique opportunities and challenges.

Characteristics of Health-related Self-report Measures for Children Aged Three to Eight Years: A Review of the Literature

AIMS

To review and make recommendations about the format and quality of health-related self-report measures for children aged 3-8 years.

METHODS

Literature searches used to identify measures of QOL, self-esteem, self-concept and mental health. The format (i.e., scale type, presentation style) and quality (i.e., item generation, reliability, validity, responsiveness) of measures were compared and evaluated.

RESULTS

Fifty three measures were identified: QOL (n = 25, 47%), self-esteem/concept (n = 15, 28%), mental health (n = 13, 25%). Likert scales were used most frequently to represent response choices (n = 34, 64%). The authors of 11 (21%) measures provided justification for their scale choice. Items were most commonly presented in written format (n = 24, 45%). Item content was generated from the respondent population in only 21 (40%) measures. Twenty-seven (51%) measures reported internal reliability between 0.70 and 0.90, and 12 (23%) reported reproducibility in this range. Although validity was reported for 48 (91%) measures, evidence for three or more aspects occurred for only 9 (17%). Eleven (21%) measures evidenced responsiveness to change.

CONCLUSION

Authors should provide clearer evidence for reliability and responsiveness. Newly developed instruments need to meet established standards, and further studies should assess the impact of scale and presentation types on the psychometrics of measures.

Different factor loadings for SF36: The Strong Heart Study and the National Survey of Functional Health Status

BACKGROUND

To increase our understanding of the psychometric characteristics and factor structure of the SF36 in older American Indian populations.

METHODS

Between 1993 and 1995, SF36 data were collected from 3,488 Phase II participants of the Strong Heart Study (SHS) between the ages of 48 and 81. Comparison data were provided by an age- and gender-matched sample (n = 695) from the National Survey of Functional Health Status (NSFHS) conducted in 1989 and 1990.

RESULTS

Generally, the basic psychometric analyses showed that the SF36 performed adequately in these older American Indians. Exploratory factor analyses indicated that a one-factor model best fit the data for both older groups. On the other hand, confirmatory factor analyses showed that a two-factor model with correlated factors provided a superior fit to the data than a one-factor model. An assumption of equivalent factor loadings for the SHS and NSFHS groups was untenable.

CONCLUSION

These analyses demonstrate that use of summary scores assuming a differentiated physical/mental functioning structure is likely improper in at least some populations. The SF36 provides an important opportunity to understand cultural differences in the conceptualization and measurement of health-related quality of life.

The structural equation modeling technique did not show a response shift, contrary to the results of the then test and the individualized approaches

BACKGROUND AND OBJECTIVE

Persons experiencing changes in their physical health may change their values and rerate the importance of basic elements of health-related quality of life (HRQL), a process known as response shift. Developing an estimator of HRQL that differentiates between objective change and response shift is essential for the interpretation of the results. The purpose of the present article was to contrast three methodologic approaches for evaluating response shift to develop a proposed set of HRQL measurement recommendations under circumstances where response shift is expected to occur.

METHODS

The three approaches compared were a structural equation modeling (SEM) technique, the then test, and an individualized approach. The data collection procedures for these methods were incorporated into a poststroke randomized controlled trial.

RESULTS

The SEM did not show a response shift, contrary to the results of the then test and the individualized approaches. We discuss factors that effect the selection of a methodologic approach including feasibility, subjects' memory and more advanced cognitive tasks, and whether response shift was evaluated at the group or individual level.

CONCLUSION

The evaluation of response shift is an integral part of HRQL evaluations, and further comparisons between methodologic approaches are needed.

Incorporating the International Classification of Functioning, Disability, and Health (ICF) into an electronic health record to create indicators of function: proof of concept using the SF-12

OBJECTIVE

The purpose of this proof-of-concept study was to assess the feasibility of using a generic health measure to create coded functional status indicators and compare the characterization of a stroke population using coded functional indicators and using health-related quality-of-life summary measures alone.

DESIGN

Multiple raters assigned International Classification of Functioning, Disability, and Health (ICF) codes to the items of the 12-Item Short Form Health Survey (SF-12). Data for comparing the information from the SF-12 and from ICF codes were derived from the Montreal Stroke Cohort Study that was set up to examine the long-term impact of stroke. Available for analysis were data from 604 persons with stroke, average age 69 years, and 488 controls, average age 62 years.

MEASUREMENT

The SF-12 provides two summary scores, one for physical health and one for mental health. Domains of the ICF are coded to three digits, before the decimal; specific categorizations of impairments, activity limitations, and participation restrictions are coded to four digits before the decimal.

RESULTS

Persons with stroke scored, on average, approximately 10 points lower than controls on physical and mental health. The ICF coding indicated that this was attributed, not surprisingly, to greater difficulty in doing moderate activities including housework, climbing stairs, and working and was not attributed to differences in pain. Differences in mental health were attributed most strongly to greater fatigue (impairment in energy), but all areas of mental health were affected to some degree.

CONCLUSION

The ICF coding provided enhanced functional status information in a format compatible with the structure of administrative health databases.

Measurement of children's health

Measuring child health has become a more pressing issue in the face of high rates of survival of childhood illnesses, often with residual chronic health conditions or impairments; the growing awareness of child behavioral and mental health morbidity; the focus on the importance of assessing therapeutic interventions and outcomes; and increased awareness of the multiple ways in which child health influences later adult health conditions. There is widespread belief that the perfect all-purpose measure can be developed and that researchers have not yet succeeded in doing so. This paper addresses three kinds of challenges to the measurement of child health that hamper the development of measures and make it likely that one size will not fit all. These include a series of fundamental underlying assumptions about health and its measurement, issues unique to assessing the health of children, and unresolved controversies that relate to the lens through which to assess child health.

Reconsidering the psychometrics of quality of life assessment in light of response shift and appraisal

The increasing evidence for response shift phenomena in quality of life (QOL) assessment points to the necessity to reconsider both the measurement model and the application of psychometric analyses. The proposed psychometric model posits that the QOL true score is always contingent upon parameters of the appraisal process. This new model calls into question existing methods for establishing the reliability and validity of QOL assessment tools and suggests several new approaches for describing the psychometric properties of these scales. Recommendations for integrating the assessment of appraisal into QOL research and clinical practice are discussed.

Advancing health disparities research: can we afford to ignore measurement issues?

BACKGROUND

Research on racial and ethnic health disparities in the United States requires that self-report measures, developed primarily in mainstream samples, are appropriate when applied in diverse groups. To compare groups, mean scores must reflect true scores and have minimal bias, assumptions that have not been tested for many self-report measures used in this research.

OBJECTIVE

To identify conceptual and psychometric issues that need to be addressed to assure the quality of self-report measures being used in health disparities research.

METHODS

We present 2 broad conceptual frameworks for health disparities research and describe the main research questions and measurement issues for 4 key concepts hypothesized as potential mechanisms of health disparities: socioeconomic status, discrimination, acculturation, and quality of care. This article is based on a small conference convened by 6 Resource Centers for Minority Aging Research (RCMAR) measurement cores. We integrate written materials prepared for the conference by quantitative and qualitative measurement specialists and cross-cultural researchers, conference discussions, and current literature.

RESULTS

Problems in the quality of the conceptualizations and measures were found for all 4 concepts, and little is known about the extent to which measures of these concepts can be interpreted similarly across diverse groups. Many problems also apply to other concepts relevant to health disparities. We propose an agenda for accomplishing this challenging measurement research.

CONCLUSIONS

The current national commitment to reduce health disparities may be compromised without more research on measurement quality. Integrated, systematic efforts are needed to move this work forward, including collaborative efforts and special initiatives.