The night out task and scoring application: an ill-structured, open-ended clinic-based test representing cognitive capacities used in everyday situations

Conceptualization of the term "ecological validity" in neuropsychological research on executive function assessment: a systematic review and call to action

OBJECTIVE

"Ecological validity" (EV) is classically defined as test's ability to predict real-world functioning, either alone or together with test's similarity to real-world tasks. In neuropsychological literature on assessment of executive functions (EF), EV is conceptualized inconsistently, leading to misconceptions about the utility of tests. The goal of this systematic review was to examine how EV is conceptualized in studies of EF tests described as ecologically valid.

METHOD

MEDLINE and PsychINFO Databases were searched. PRISMA guidelines were observed. After applying inclusion and exclusion criteria, this search yielded 90 articles. Deductive content analysis was employed to determine how the term EV was used.

RESULTS

About 1/3 of the studies conceptualized EV as the test's ability to predict functional outcomes, 1/3 as both the ability to predict functional outcome and similarity to real-world tasks, and 1/3 were either unclear about the meaning of the term or relied on notions unrelated to classical definitions (e.g., similarity to real-world tasks alone, association with other tests, or the ability to discriminate between populations).

CONCLUSIONS

Conceptualizations of the term EV in literature on EF assessment vary grossly, subsuming the notions of criterion, construct, and face validity, as well as sensitivity/specificity. Such inconsistency makes it difficult to interpret clinical utility of tests that are described as ecologically valid. We call on the field to require that, at minimum, the term EV be clearly defined in all publications, or replaced with more concrete terminology (e.g., criterion validity).

Six elements test vs D-KEFS: what does "Ecological Validity" tell us?

OBJECTIVE

Extensive research shows that tests of executive functioning (EF) predict instrumental activities of daily living (IADLs) but are nevertheless often criticized for having poor ecological validity. The Modified Six Elements Test (MSET) is a pencil-and-paper test that was developed to mimic the demands of daily life, with the assumption that this would result in a more ecologically valid test. Although the MSET has been extensively validated in its ability to capture cognitive deficits in various populations, support for its ability to predict functioning in daily life is mixed. This study aimed to examine the MSET's ability to predict IADLs assessed via three different modalities relative to traditional EF measures.

METHOD

Participants (93 adults aged 60 - 85) completed the MSET, traditional measures of EF (Delis-Kaplan Executive Function System; D-KEFS), and self-reported and performance-based IADLs in the lab. Participants then completed three weeks of IADL tasks at home, using the Daily Assessment of Independent Living and Executive Skills (DAILIES) protocol.

RESULTS

The MSET predicted only IADLs completed at home, while the D-KEFS predicted IADLs across all three modalities. Further, the D-KEFS predicted home-based IADLs beyond the MSET when pitted against each other, whereas the MSET did not contribute beyond the D-KEFS.

CONCLUSIONS

Traditional EF tests (D-KEFS) appear to be superior to the MSET in predicting IADLs in community-dwelling older adults. The present results argue against replacing traditional measures with the MSET when addressing functional independence of generally high-functioning and cognitive healthy older adult patients.

A Clinic-Based Measure of Everyday Planning Ability: The Overnight Trip Task

OBJECTIVE

Traditional measures of cognitive abilities most used by neuropsychologists are generally low in ecological validity and only capture a small proportion of the variance of a person's true functional capacity. This study evaluates the association between clinic-based performance and parallel real-world completion of an everyday planning test, the Overnight Trip Task (OTT).

METHOD

A cross-sectional cohort of 65 community-dwelling older adults completed a battery of cognitive assessments and two formats of the OTT: the paper-and-pencil clinic OTT, which was completed remotely through video conferencing (Zoom), and the parallel real-world OTT (RW-OTT), which was completed at home between sessions. Both formats required participants to plan for what they would pack and prepare for a hypothetical overnight trip based on a provided story that included rules and embedded contingencies.

RESULTS

The clinic OTT demonstrated a small-to-moderate relationship with the RW-OTT (r = 0.35) and no relationship with measures of learning, long delay recall, an executive function composite, and a measure of everyday planning. Hierarchical regressions indicated that the clinic OTT demonstrated incremental validity above an executive function composite measure and global cognition when predicting self-reported everyday functioning and RW-OTT performance.

CONCLUSIONS

The clinic OTT showed only modest association with the RW-OTT and discriminate but not convergent validity was demonstrated. The clinic OTT showed incremental validity when predicting self-reported everyday functioning and RW-OTT performance above more traditional measures. Before the clinic OTT could be considered a reliable and valid clinic-based measure for predicting real-world behavior, additional research would be needed.

A Multidimensional, Person-Centered Framework for Functional Assessment in Dementia: Insights from the 'What', 'How', 'To Whom', and 'How Much' Questions

Dementia is a syndrome characterized by cognitive and neuropsychiatric symptoms associated with progressive functional decline (FD). FD is a core diagnostic criterion for dementia, setting the threshold between its prodromal stages and the full-blown disease. The operationalization of FD continues to generate a great deal of controversy. For instance, the threshold of FD for the diagnosis of dementia varies across diagnostic criteria, supporting the need for standardization of this construct. Moreover, there is a need to reconsider how we are measuring FD to set boundaries between normal aging, mild cognitive impairment, and dementia. In this paper, we propose a multidimensional framework that addresses outstanding issues in the assessment of FD: i) What activities of daily living (ADLs) are necessary to sustain an independent living in aging? ii) How to assess FD in individuals with suspected neurocognitive disorders? iii) To whom is the assessment directed? and iv) How much does FD differentiate healthy aging from mild and major neurocognitive disorders? Importantly, the To Whom Question introduces a person-centered approach that regards patients and caregivers as active agents in the assessment process of FD. Thus, once impaired ADLs have been identified, patients can indicate how significant such impairments are for them in daily life. We envisage that this new framework will guide future strategies to enhance functional assessment and treatment of patients with dementia and their caregivers.

It's complicated: Executive functioning moderates impacts of daily busyness on everyday functioning in community-dwelling older adults

OBJECTIVE

Research shows that cognitively healthy older adults with mild executive function (EF) weaknesses are vulnerable to the negative impacts of life complexity (or daily busyness) when performing instrumental activities of daily living (IADLs). However, past research assessed life complexity only at one timepoint, not capturing daily fluctuations. Importantly, fluctuations in busyness can themselves have deleterious impacts on functioning. This study extended past research by examining whether (1) variability in daily busyness would be more detrimental than level of busyness to performance of IADLs, and (2) EF assessed at home would moderate deleterious impact of busyness on IADLs.

METHOD

Fifty-two community-dwelling older adults aged 60 to 95 completed daily IADL tasks and daily measures of EF and busyness via ecological momentary assessment, independently at home for 18 days.

RESULTS

(1) In a subset of participants with mild EF weaknesses, high variability in busyness across days was associated with fewer tasks completed correctly; and (2) across all participants (regardless of EF), high levels of daily busyness were associated with fewer tasks completed on time.

CONCLUSIONS

Findings indicate that high variability in daily busyness, potentially reflecting a lack of daily routine, was associated with IADL errors among cognitively healthy older adults with mild EF weaknesses. Additionally, consistently high levels of busyness were associated with failures to complete tasks, or failures to complete them on time, regardless of EF. These results further support the Contextually Valid Executive Assessment (ConVExA) model, which posits that EF and contextual factors interact to predict functional outcomes.

Assessing functional ability of healthy adults with the Night Out Task

Objective: Given the negative health outcomes associated with functional loss, there is need to better understand the trajectory of functional change and compensation use with age. Many older adults successfully age in place, and there is not a one-to-one relationship between cognitive changes and ability to live independently. This study compared healthy age cohorts' performance and approach on a functional measure, particularly examining compensatory strategy use to support task performance. Methods: 57 young adults (YAs; ages 18-39), 42 young-old adults (YOAs; ages 60-69) and 47 old-old (OOAs; ages 70+) completed the Night Out Task (NOT); an open-ended functional assessment in which participants complete eight subtasks in preparation for a night out with a friend (e.g. prepare tea and gather items for a recipe). The NOT measures both primary outcome variables (e.g. time and accuracy), error types and 'compensation variables', which are process-approach variables intended to map on to the types of compensatory strategies individuals use in their everyday lives (e.g. planning and checking). Results: Results revealed that YAs performed better than the oldest group on the NOT in accuracy, execution time and number of both inefficient and inaccurate/incomplete errors. YAs additionally used fewer compensatory strategies than both older groups. Only one compensation variable had a positive relationship to outcome; double-checking was related to improved accuracy and fewer inaccurate/incomplete errors within the oldest cohort. Conclusion: Together these findings support a spectrum of functional change with age. While compensation use increased with age, the relationship between compensation use and outcome was unclear. Future work is needed to understand under what conditions older adults' self-initiate compensation use and to understand the relationship between compensation use and outcome.

Ecological validity reconsidered: the Night Out Task versus the D-KEFS

INTRODUCTION

Although executive functioning (EF) correlates with execution of instrumental activities of daily living (IADLs), tests of EF have been criticized for having poor ecological validity. Attempts have been made to develop new tests that approximate naturalistic daily tasks. However, the incremental utility of such tests has not been convincingly demonstrated. The Night Out Task (NOT) is a novel measure designed to increase ecological validity. This study examined whether the NOT correlates with traditional lab- and home-based measures of EF and IADLs, and whether it outperforms traditional measures of EF in predicting IADLs.

METHOD

Participants (50 adults aged 60 to 95) completed (1) the Delis Kaplan Executive Function System (D-KEFS) and IADLs in the laboratory, and (2) ecological momentary assessment of EF and daily IADL tasks at home across three weeks (using the Daily Assessment of Independent Living and Executive Skills protocol; DAILIES).

RESULTS

The NOT correlated with a lab-based measure of EF beyond covariates, and lab-based IADLs beyond covariates and beyond the D-KEFS. However, it was unrelated to at-home variables beyond covariates. In contrast, the D-KEFS was a significant predictor of at-home IADLs, and this association was mediated by at-home EF performance.

CONCLUSION

This study provides a preliminary validation of the NOT as a correlate of office-based performances in a primarily college educated white sample. Despite its high face validity, the NOT does not appear to sufficiently tap EF processes needed for home-based IADLs as measured by the DAILIES, although small sample size limits the interpretability of this negative finding.

Methodological Issues in Advancing the Status of Functional Cognitive Assessment

Many individuals in acute hospital and post-acute care settings experience changes in their capacity to perform complex activities of daily living associated with deficits in functional cognition. Occupational therapists regularly assess and treat these occupational performance deficits. The construct of functional cognition offers oportunities for occupational therapists to define an approach to cognition that is both distinct from that of other disciplines and that supports evidence-based interventions. This article provides a rationale for performance-based assessment of functional cognition and an overview of the methodological issues associated with the development and implementation of reliable and valid screening and comprehensive asseements of functional.