Naturalistic tasks performed in realistic environments: a review with implications for neuropsychological assessment

Six elements test vs D-KEFS: what does "Ecological Validity" tell us?

OBJECTIVE

Extensive research shows that tests of executive functioning (EF) predict instrumental activities of daily living (IADLs) but are nevertheless often criticized for having poor ecological validity. The Modified Six Elements Test (MSET) is a pencil-and-paper test that was developed to mimic the demands of daily life, with the assumption that this would result in a more ecologically valid test. Although the MSET has been extensively validated in its ability to capture cognitive deficits in various populations, support for its ability to predict functioning in daily life is mixed. This study aimed to examine the MSET's ability to predict IADLs assessed via three different modalities relative to traditional EF measures.

METHOD

Participants (93 adults aged 60 - 85) completed the MSET, traditional measures of EF (Delis-Kaplan Executive Function System; D-KEFS), and self-reported and performance-based IADLs in the lab. Participants then completed three weeks of IADL tasks at home, using the Daily Assessment of Independent Living and Executive Skills (DAILIES) protocol.

RESULTS

The MSET predicted only IADLs completed at home, while the D-KEFS predicted IADLs across all three modalities. Further, the D-KEFS predicted home-based IADLs beyond the MSET when pitted against each other, whereas the MSET did not contribute beyond the D-KEFS.

CONCLUSIONS

Traditional EF tests (D-KEFS) appear to be superior to the MSET in predicting IADLs in community-dwelling older adults. The present results argue against replacing traditional measures with the MSET when addressing functional independence of generally high-functioning and cognitive healthy older adult patients.

Bridging the Gap Between Self-Report and Behavioral Laboratory Measures: A Real-Time Driving Task With Inverse Reinforcement Learning

A major challenge in assessing psychological constructs such as impulsivity is the weak correlation between self-report and behavioral task measures that are supposed to assess the same construct. To address this issue, we developed a real-time driving task called the "highway task," in which participants often exhibit impulsive behaviors mirroring real-life impulsive traits captured by self-report questionnaires. Here, we show that a self-report measure of impulsivity is highly correlated with performance in the highway task but not with traditional behavioral task measures of impulsivity (47 adults aged 18-33 years). By integrating deep neural networks with an inverse reinforcement learning (IRL) algorithm, we inferred dynamic changes of subjective rewards during the highway task. The results indicated that impulsive participants attribute high subjective rewards to irrational or risky situations. Overall, our results suggest that using real-time tasks combined with IRL can help reconcile the discrepancy between self-report and behavioral task measures of psychological constructs.

Principles for Adapting Assessments of Executive Function across Cultural Contexts

Direct assessments of executive functions (EFs) are increasingly used in research and clinical settings, with a central assumption that they assess "universal" underlying skills. Their use is spreading globally, raising questions about the cultural appropriateness of assessments devised in Western industrialized countries. We selectively reviewed multidisciplinary evidence and theory to identify sets of cultural preferences that may be at odds with the implicit assumptions of EF assessments. These preferences relate to motivation and compliance; cultural expectations for interpersonal engagement; contextualized vs. academic thinking; cultural notions of speed and time; the willingness to be silly, be incorrect, or do the opposite; and subject-matter familiarity. In each case, we discuss how the cultural preference may be incompatible with the assumptions of assessments, and how future research and practice can address the issue. Many of the cultural preferences discussed differ between interdependent and independent cultures and between schooled and unschooled populations. Adapting testing protocols to these cultural preferences in different contexts will be important for expanding our scientific understanding of EF from the narrow slice of the human population that has participated in the research to date.

Insights gained over 60 years on factors shaping post-stroke aphasia recovery: A commentary on Vignolo (1964)

Aphasia is an acquired language disorder resulting from brain injury, including strokes which is the most common etiology, neurodegenerative diseases, tumors, traumatic brain injury, and resective surgery. Aphasia affects a significant portion of stroke survivors, with approximately one third experiencing its debilitating effects in the long term. Despite its challenges, there is growing evidence that recovery from aphasia is possible, even in the chronic phase of stroke. Sixty years ago, Vignolo (1964) outlined the primary challenges confronted by researchers in this field. These challenges encompassed the absence of an objective evaluation of language difficulties, the scarcity of evidence regarding spontaneous aphasia recovery, and the presence of numerous variables that could potentially influence the process of aphasia recovery. In this paper, we discuss the remarkable progress that has been made in the assessment of language and communication in aphasia as well as in understanding the factors influencing post-stroke aphasia recovery.

Translation, cross-cultural adaptation and validation of the 10-item Weekly Calendar Planning Activity in Spanish-speaking ABI patients: a multicenter study

We present the process of translation, adaptation, and validation in the Spanish context of the 10-item version of the Weekly Calendar Planning Activity (WCPA-10), a performance-based measure of cognitive instrumental activities of daily living (C-IADL). The study consisted of two phases: I) translation/cultural adaptation of the WCPA, conducted by professional bilingual translators, a panel of experts, and a pilot study, and II) validation in a sample of 42 acquired brain injury patients (ABI) and 42 healthy participants (HC). WCPA primary outcomes showed expected convergent/discriminant validity patterns with socio-demographical and clinical variables and cognitive processes identifying those WCPA outcomes that best predicted executive and memory deficits measured with a battery of traditional neuropsychological tests. In addition, performance on the WCPA was a significant predictor of everyday functioning over variables such as socio-demographics or global cognition when measured with traditional tests. External validity was established by the WCPA's ability to identify everyday cognitive deficits in ABI patients compared to HC, even in those with subtle cognitive impairment based on neuropsychological tests. The Spanish WCPA-10 seems an appropriate and sensitive assessment tool to identify cognitive-functional impairment in ABI-patients, even those with subtle cognitive impairment. The results also highlight the relevance of this kind of test, as they indicate a better prediction of patients' real-world functioning than traditional neuropsychological tests.

Characterizing Cognition in Everyday Life of Older Adults With Subjective Cognitive Decline

Subjective cognitive decline (SCD), the subjective experience of worsening cognition with no objective cognitive impairment, poses a heightened risk for dementia. This study aimed to characterize cognition in the everyday life of people with SCD, is crucial for understanding and preventing further functional and cognitive decline. One hundred and thirty-five older adults (age 70.7±6.7) with SCD were assessed using functional-cognition measures: Multifactorial Memory Questionnaire (MMQ), Behavior Rating Inventory of Executive Function–Adult version (BRIEF-A), and Multiple Errands Test (MET). The resulted showed that older adults with SCD reported lower memory satisfaction (Hedges’s g = 0.41) on the MMQ, and worse metacognition on the BRIEF-A (Hedges’s g = 0.63) compared with published normative data. They completed an average of only 6/12 required tasks on the MET. The findings show functional difficulties related to SCD and inform the development of occupational therapy intervention for this population.

Initial Validation of the Weekly Calendar Planning Activity With Adolescents With ABI

The purpose of this study was to assess the validity of the Weekly Calendar Planning Activity Middle/High School (WCPA) in adolescents with acquired brain injuries (ABI). We recruited neurotypical controls (n = 27) and adolescents with ABI (n = 14) to complete a 90-min battery of cognitive tests. Adolescents with ABI were further divided into mild and severe groups by Glasgow Coma Scale Score and compared with controls. Mann-Whitney U tests revealed significant differences (p = .05) on the WCPA between groups in total time to complete and rules followed. Between the controls and severe group, a significant difference (p = .05) was found for total time spent, errors, rules followed, strategies used, and accurate appointments (p = .01). Moderate correlations were found between the WCPA and neuropsychological tests (ρ = .31-.45). The WCPA is a valid performance-based assessment of functional cognition that can be used to detect deficits in adolescents with severe ABI.

Daily Activities: The Impact of COPD and Cognitive Dysfunction

OBJECTIVE

Chronic obstructive pulmonary disease (COPD) is characterized by airflow limitation; however, pulmonary function does not fully account for patients' functional difficulties. The primary aim of the study was to determine the association between several domains of cognition and daily activity among those with COPD.

METHOD

Eighty-nine former smokers completed a neuropsychological battery including measures across multiple domains of cognition, pulmonary function measures, and daily activity questionnaires. Using a cross-sectional design, we compared daily activity between former smokers with and without COPD using two measures (St. George's Respiratory Questionnaire [SGRQ] Activity Subscale and Lawton Instrumental Activities of Daily Living [IADL] Scale) and examined the association between cognition and daily activity among those with COPD.

RESULTS

As expected, former smokers with COPD reported more difficulty than those without COPD on both activity measures (SGRQ Activity Subscale p < .001; Lawton IADL Scale p = .040). Among former smokers with COPD, poorer delayed recall was associated with more difficulty with daily activities (SGRQ Activity Subscale) (p = .038) while adjusting for severity of airflow limitation, exercise tolerance, oxygen use, dyspnea, and symptoms of anxiety and depression.

CONCLUSION

The findings suggest that cognition is associated with daily activity in patients with COPD. Future research should examine whether cognitive interventions may help to maximize patients' engagement in daily activities.

The night out task and scoring application: an ill-structured, open-ended clinic-based test representing cognitive capacities used in everyday situations

OBJECTIVE

The night out task (NOT) was developed as a naturalistic, open-ended, multitasking measure that requires individuals to complete eight subtasks comparable to those encountered during real-world functioning (e.g., pack travel bag, prepare tea). We examined psychometric properties and administration feasibility of this direct observation measure within a clinic-like setting using a tablet-based coding application.

METHOD

A sample of 148 community-dwelling older adults (82% cognitively healthy; 18% mild cognitive impairment) and 57 younger adults completed the NOT along with other neurocognitive tests and questionnaires.

RESULTS

Inter-rater reliability across NOT primary (i.e., time, accuracy, efficiency) and process-related (e.g., error-types, self-corrections) variables was mostly excellent. NOT primary measures showed expected patterns of convergent and discriminant validity with measures of cognition, demographics, and well-being. External validity was established by the NOT ability to distinguish between age and diagnostic (cognitively healthy vs. mild cognitive impairment) groups. Demonstrating incremental validity, the NOT primary variables (execution time in particular) were predictive of self-reported functional abilities and completion quality of in-home everyday tasks over and earlier variables such as demographics, cognition, and mobility.

CONCLUSIONS

These findings suggest that the NOT and its app interface, which allows for continuous logging of observations, are a feasible in-clinic measure to assess cognitive capacities important for real-world functioning. With further validation, the NOT may allow for earlier detection of functional difficulties. Understanding errors and strategies used during NOT performance could also have implications for individualized interventions.

Cognition and daily life functioning among persons with serious mental illness: A cluster analytic examination of heterogeneity on the Test of Grocery Shopping Skills

OBJECTIVE

To characterize variability in daily life functioning among individuals with serious mental illness based on a naturalistic performance measure of grocery shopping and standard neuropsychological tasks using cluster analytic methods.

METHODS

A naturalistic performance measure, the Test of Grocery Shopping Skills (TOGSS), and standard neuropsychological tasks, were completed by 191 participants with serious mental illness. Hierarchical cluster analytic techniques were used to explore functional subgroups based on naturalistic performance measure variables. Multivariate analyses of variance were utilized to compare subgroups on TOGSS variables and neuropsychological measures, respectively.

RESULTS

Two distinct functional subgroups emerged from the cluster analysis. On average, participants in cluster one were faster, more efficient, and more accurate compared to cluster two. Based on performance on neuropsychological tasks, cluster one had better verbal memory, visual attention, and processing speed, and executive functioning scores, compared to cluster two. The clusters did not differ on a measure of auditory working memory.

CONCLUSIONS AND IMPLICATIONS FOR PRACTICE

Naturalistic performance measures can assist with characterizing the heterogeneity in real life functioning among people with serious mental illness. Further work to illuminate the relationship between specific cognitive abilities and specific functional abilities is warranted and may assist with targeting effective treatment plans for functional recovery. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Marrying Past and Present Neuropsychology: Is the Future of the Process-Based Approach Technology-Based?

A cognitive assessment strategy that is not limited to examining a set of summary test scores may be more helpful for early detection of emergent illness such as Alzheimer's disease (AD) and may permit a better understanding of cognitive functions and dysfunctions in those with AD and other dementia disorders. A revisit of the work already undertaken by Kaplan and colleagues using the Boston Process-Approach provides a solid basis for identifying new opportunities to capture data on neurocognitive processes, test-taking strategies and response styles. Thus, this critical review will combine traditional process-based assessment strategies with support provided or offered by newer technologies that have the potential to add value to data collection and interpretation. There is now considerable interest in neuropsychological test administration using computer/digital technology, both in research and in clinical settings. To add value, any computerized version of an existing cognitive test should respect the administration procedure for which normative data were obtained, should be time-saving in terms of scoring and interpretation, and should, we argue, facilitate gathering information about the processes and strategies followed in test completion. This article will offer an overview of the steps needed when implementing computerization of neuropsychological tests using a Process-Based Approach (PBA) to these technology-based adaptations and will discuss further developments in this area by linking it to future technological developments that may be possible in the area of neuropsychological assessment. Additionally, an overview of neuropsychological tests that may benefit from computerization will be presented, together with suggestions on the specific processes, strategies and features that may be captured with the aid of such computerization. Finally, hypotheses on how virtual reality could be an asset for the future of the PBA to neuropsychological assessment will also be discussed.

Assessment of cognitive instrumental activities of daily living: a systematic review

PURPOSE

Cognitive instrumental activities of daily living are particularly related to executive functions, such as scheduling appointments, monthly payments, managing the household economy, shopping or taking the bus. The aim of this systematic review was to determine the available tests for the assessment of executive functions with ecological validity to predict individuals' functioning.

MATERIALS AND METHODS

An electronic search was conducted in MEDLINE, Cochrane Central, PsyCInfo and IEEE Xplore until May 2019, in addition to a manual search. The PRISMA criteria and the Covidence platform were used to select articles and extract data.

RESULTS

After applying the search selection criteria, 76 studies were identified. They referred to 110 tools to assess instrumental activities of daily living. Those that have received most attention are related to menu preparation and shopping. Performance-based measures are the most widely used traditional methods. Most tests were aimed at the adult population with acquired brain damage, cognitive impairment or dementia. There was a predominance of tests based on the Multiple Errands Test paradigm.

CONCLUSIONS

In recent years, it has increased the number of tools that assess the instrumental activities of daily living based on technologies such as personal or environmental sensors and serious games.IMPLICATIONS FOR REHABILITATIONAssessment of Instrumental Activities of Daily Living through performance-based measures is especially useful for the early detection of dysfunctions or preclinical disability.Difficulties in performing instrumental activities of daily living are closely associated with deficits in executive functions and prospective memory.Activities of Daily Living can be understood as multitasks.The use of virtual reality-based tests was shown to be sensitive to the detection of cognitive deficits in Activities of Daily Living.An advantage of using virtual reality in assessments is that it can help to predict the level of personal autonomy in patients who are in an institutional environment and could be a first approximation to the real environment.

Relationships among functional capacity, cognition, and naturalistic skill performance in people with serious mental illness

There has been increasing interest in understanding real-world outcomes for individuals with serious mental illnesses (SMI). This study examined domain-specific skill knowledge, functional capacity, and neurocognition as predictors of naturalistic grocery shopping skill performance in forty-eight individuals with SMI. Participants completed measures of skill knowledge and general functional capacity (UCSD Performance-Based Skill Assessment - brief) as well as measures of neurocognition and symptoms. The Test of Grocery-Shopping Skills (TOGSS) assessed naturalistic shopping. TOGSS was significantly correlated with functional capacity, shopping skill knowledge, and neurocognition, but not symptoms. Multiple regression analysis was conducted with variables entered in 2 blocks. Skill knowledge and functional capacity were entered in block 1. Neurocognitive measures were entered in block 2 using forward entry. Skill knowledge was not a significant predictor of TOGSS when accounting for functional capacity and neurocognition. Functional capacity predicted TOGSS over and above skill knowledge and remained significant after accounting for the effects of neurocognition. Our findings indicate that functional capacity was associated with skill assessment under naturalistic conditions. Further, there was some, but not complete, overlap between neurocognitive and functional capacity predictors of naturalistic skill performance. Further development of naturalistic assessments may hold promise for interventions targeting real-world function.

Naturalistic Assessment using a Simulated Environment: Cognitive Correlates and Relationship to Functional Status in Individuals with Neurologic Conditions

Objective

Research has shown that neurologic conditions, such as traumatic brain injury and multiple sclerosis, result in a number of cognitive and functional deficits. However, little is known about the relationship between various cognitive domains and ability to perform everyday activities. The Community Shopping Task (CST), a naturalistic assessment task conducted in a simulated environment, was used to examine functional abilities and cognitive correlates of everyday functioning in individuals with neurologic conditions.

Method

Thirty-four participants with neurologic conditions and 34 healthy controls completed the CST as well as traditional paper-pencil measures of cognition. In addition, all participants completed a questionnaire assessing instrumental activities of daily living (IADLs).

Results

The results indicated that participants with neurologic conditions required significantly more cues and time to complete the CST compared to control participants and that immediate memory and executive functioning were important predictors of CST performance. Furthermore, time to complete the CST accounted for a significant amount of variance in IADL performance, over and beyond the traditional measures of cognition.

Conclusions

These results provide evidence that a naturalistic task completed in an everyday environment can enhance our understanding of how daily functioning is impacted in individuals with neurologic conditions and subsequently inform rehabilitation strategies.

Electrophysiological approaches in the study of cognitive development outside the lab

The use of human neuroimaging technology provides knowledge about several emotional and cognitive processes at the neural level of organization. In particular, electroencephalographic (EEG) techniques allow researchers to explore high-temporal resolution of the neural activity that underlie the dynamics of cognitive processes. Although EEG research has been mostly applied in laboratory settings, recently a low-cost, portable EEG apparatus was released, which allows exploration of different emotional and cognitive processes during every-day activities. We compared a wide range of EEG measures using both a low-cost portable and a high-quality laboratory system. EEG recordings were done with both systems while participants performed an active task (Go/NoGo) and during their resting-state. Results showed similar waveforms in terms of morphology and amplitude of the ERPs, and comparable effects between conditions of the applied Go/NoGo paradigm. In addition, the contribution of each frequency to the entire EEG was not significantly different during resting-state, and fluctuations in amplitude of oscillations showed long-range temporal correlations. These results showed that low-cost, portable EEG technology can provide an alternative of enough quality for measuring brain activity outside a laboratory setting, which could contribute to the study of different populations in more ecological contexts.

Comparison of performance-based assessment and real world skill in people with serious mental illness: Ecological validity of the Test of Grocery Shopping Skills

Valid functional measures are essential for clinical and research efforts that address recovery and community functioning in people with serious mental illness. Although there is a great deal of interest in functional assessment, there is limited research supporting how well current evaluation methods provide a true assessment of real world functioning or naturalistic behavior. To address this gap in the literature, the present study examined the performance of individuals with serious mental illness (i.e., diagnosis of schizophrenia-spectrum, bipolar disorder, or other depression/anxiety diagnoses and accompanying functional disability) on the Test of Grocery Shopping Skills (TOGSS), a performance-based naturalistic task. We compared TOGSS performance to two dimensions of real world functioning: directly observed real world grocery shopping and ratings of community functioning. Results indicated that the TOGSS was significantly associated with real life grocery shopping, in terms of both shopping accuracy (r = 0.424) and time (r = 0.491). Further, self-report and observer-rated methods of assessing real world shopping behaviors were significantly correlated (r = 0.455). To our knowledge, this is one of the first studies to directly compare a performance-based naturalistic skill assessment with carefully observed real world performance of that skill in people with serious mental illness. These findings support the feasibility and ecological validity of performance-based naturalistic assessment with the TOGSS.