Understanding the ecological validity of neuropsychological testing using an ethnographic approach

Ecological validity in neurocognitive assessment: Systematized review, content analysis, and proposal of an instrument

OBJECTIVES

The main objectives of this study are to identify the dimensions of Ecological Validity (EV) within the definitions of this concept, understand how they are operationalized in neurocognitive tests, and propose a checklist for EV attributes in neurocognitive tests.

METHOD

A systematized review was combined with content analysis of the selected papers, using the inductive method. We analyzed 82 studies on the EV of neurocognitive tests, 19 literature reviews and 63 empirical studies. Based on this review, we identified the relevant criteria for evaluating EV.

RESULTS

EV is a multidimensional concept with two main dimensions: representativeness and generalization. Representativeness involves the subdimensions simplicity-complexity and artificial-natural and several criteria organized on a continuum from low EV to high EV. Generalization is dependent on representativeness and is influenced by different cognitive and non-cognitive factors. We propose six stages for operationalizing EV, from defining the objectives of the neurocognitive assessment to the methodology for scoring and interpreting the results.

CONCLUSION

This systematized review helps to operationalize the concept of EV by providing a tool for evaluating and improving EV while developing new tests. Further studies with a longitudinal design can compare the predictive value of tests with higher versus lower EV-checklist scores.Key pointsQuestion: Understand the definition of EV, its dimensions and subdimensions, how EV is operationalized in neurocognitive tests and propose a checklist for the EV attributes of neurocognitive tests.Findings: The primary findings were that representativeness and generalization are the main dimensions of EV. Representativeness involves several subdimensions, whereas generalization is dependent on representativeness and is influenced by cognitive and non-cognitive factors. We provided an EV-checklist organized into six parts.Importance: The EV-checklist can be used to guide the development of ecologically valid neurocognitive tests and/or assess the EV of existing ones.Next steps: Examine the predictive value of tests that have higher EV-checklist scores.

Comparison of performance-based assessment and real world skill in people with serious mental illness: Ecological validity of the Test of Grocery Shopping Skills

Valid functional measures are essential for clinical and research efforts that address recovery and community functioning in people with serious mental illness. Although there is a great deal of interest in functional assessment, there is limited research supporting how well current evaluation methods provide a true assessment of real world functioning or naturalistic behavior. To address this gap in the literature, the present study examined the performance of individuals with serious mental illness (i.e., diagnosis of schizophrenia-spectrum, bipolar disorder, or other depression/anxiety diagnoses and accompanying functional disability) on the Test of Grocery Shopping Skills (TOGSS), a performance-based naturalistic task. We compared TOGSS performance to two dimensions of real world functioning: directly observed real world grocery shopping and ratings of community functioning. Results indicated that the TOGSS was significantly associated with real life grocery shopping, in terms of both shopping accuracy (r = 0.424) and time (r = 0.491). Further, self-report and observer-rated methods of assessing real world shopping behaviors were significantly correlated (r = 0.455). To our knowledge, this is one of the first studies to directly compare a performance-based naturalistic skill assessment with carefully observed real world performance of that skill in people with serious mental illness. These findings support the feasibility and ecological validity of performance-based naturalistic assessment with the TOGSS.

Cognitive functioning and employment among people with schizophrenia in vocational rehabilitation

BACKGROUND

Employment is central to recovery in schizophrenia, but little attention has been paid to its relationship with cognitive functioning.

OBJECTIVE

This cross-sectional study adds to the knowledge base of relationships between cognitive functioning and gaining competitive employment, work hours per week, and monthly income among people with schizophrenia in vocational rehabilitation. It also examines which area of cognitive function may be decisive for gaining employment.

METHODS

Thirty-nine vocational rehabilitation participants were administered a cognitive battery based on MATRICS Consensus Cognitive Battery. Socio-demographic, clinical, and vocational data were gathered and analyzed with nonparametric statistics.

RESULTS

Individuals with competitive employment differed from those without competitive employment in attention and psychomotor speed, delayed verbal recall, immediate visual recall, and planning, reasoning, and problem-solving. Higher scores in immediate and delayed verbal recall and planning, reasoning, and problem-solving correlated with more work hours per week and higher income. Immediate visual recall was related to higher income. Higher scores in planning, reasoning, and problem-solving was an indicator of competitive employment (OR = 1.48).

CONCLUSIONS

Higher order cognitive functioning of planning, reasoning, and problem-solving may have a central role in gaining employment. The findings should be considered in compensation for or improving cognitive functions for vocational rehabilitation participants.

Ecological validity of pediatric neuropsychological measures: current state and future directions

Neuropsychologists are increasingly requested to identify specific deficits in cognitive abilities and determine the ways in which these deficits will affect a client's everyday functioning. The demand for prescriptive diagnostic endeavors that provide recommendations for rehabilitation has drawn attention to the necessity of considering the ecological validity of instruments. Most of the current knowledge regarding the ecological validity of neuropsychological measures is the result of studies examining adult and older adult populations. The relationship of test performance and everyday functioning in children has been less examined and is, therefore, less understood. The purpose of this article is to provide a brief history of ecological validity in neuropsychology, discuss why this is an important consideration when working with child populations, and provide suggestions for continued research in this field.

The politics of action research: "if you don't like the way things are going, get off the bus"

Participatory action research is lauded for its emancipatory potential and practice relevance. Little research has focused on the ethical and political dimensions of power relations between management gatekeepers and staff in large organizations in the negotiation of action research. To illuminate the dynamics of this engagement, this paper is informed by Aristotle's practical ethics of phronesis, arguing that ethics emerge through human interaction. The paper reports on a multi-method study and intervention in inter-professional learning and collaboration, conducted between 2008 and 2010, across an Australian health system. It draws on results from researchers' progress notes, 139 semi-structured interviews and focus groups (492 participants), more than 200 h of organizational document analysis, and more than 200 h of ethnographic observation. From conventional thematic analysis, we note that the project had considerable support from senior managers who were gatekeepers for the research. Such support stemmed from managerialist tendencies, with senior managers explicitly aligning inter-professionalism with prospective health reforms and improvements to quality of care and patient safety. Senior managers were also enthusiastic about standardized processes. Many frontline staff were less supportive, and some were suspicious of or hostile towards management-led processes to improve and evaluate care. Some senior managers' self perceived alignment with and support for the research process changed to resistance once this finding was presented. This paradox in the interplay of research findings and research process evinces the inherent tension between organizational politics and conducting action research that reflexive researchers need to negotiate in knowledge translation, exchange or mobilization exercises.