Development and validation of an episodic memory measure in the Mobile Toolbox (MTB): Arranging Pictures

INTRODUCTION

Arranging Pictures is a new episodic memory test based on the NIH Toolbox (NIHTB) Picture Sequence Memory measure and optimized for self-administration on a personal smartphone within the Mobile Toolbox (MTB). We describe evidence from three distinct validation studies.

METHOD

In Study 1, 92 participants self-administered Arranging Pictures on study-provided smartphones in the lab and were administered external measures of similar and dissimilar constructs by trained examiners to assess validity under controlled circumstances. In Study 2, 1,021 participants completed the external measures in the lab and self-administered Arranging Pictures remotely on their personal smartphones to assess validity in real-world contexts. In Study 3, 141 participants self-administered Arranging Pictures remotely twice with a two-week delay on personal iOS smartphones to assess test-retest reliability and practice effects.

RESULTS

Internal consistency was good across samples (ρxx = .80 to .85, p < .001). Test-retest reliability was marginal (ICC = .49, p < .001) and there were significant practice effects after a two-week delay (ΔM = 3.21 (95% CI [2.56, 3.88]). As expected, correlations with convergent measures were significant and moderate to large in magnitude (ρ = .44 to .76, p < .001), while correlations with discriminant measures were small (ρ = .23 to .27, p < .05) or nonsignificant. Scores demonstrated significant negative correlations with age (ρ = -.32 to -.21, p < .001). Mean performance was slightly higher in the iOS compared to the Android group (MiOS = 18.80, NiOS = 635; MAndroid = 17.11, NAndroid = 386; t(757.73) = 4.17, p < .001), but device type did not significantly influence the psychometric properties of the measure. Indicators of potential cheating were mixed; average scores were significantly higher in the remote samples (F(2, 850) = 11.415, p < .001), but there were not significantly more perfect scores.

CONCLUSION

The MTB Arranging Pictures measure demonstrated evidence of reliability and validity when self-administered on personal device. Future research should examine the potential for cheating in remote settings and the properties of the measure in clinical samples.

Using SCENTinel® to predict SARS-CoV-2 infection: insights from a community sample during dominance of Delta and Omicron variants

Introduction

Based on a large body of previous research suggesting that smell loss was a predictor of COVID-19, we investigated the ability of SCENTinel®, a newly validated rapid olfactory test that assesses odor detection, intensity, and identification, to predict SARS-CoV-2 infection in a community sample.

Methods

Between April 5, 2021, and July 5, 2022, 1,979 individuals took one SCENTinel® test, completed at least one physician-ordered SARS-CoV-2 PCR test, and endorsed a list of self-reported symptoms.

Results

Among the of SCENTinel® subtests, the self-rated odor intensity score, especially when dichotomized using a previously established threshold, was the strongest predictor of SARS-CoV-2 infection. SCENTinel® had high specificity and negative predictive value, indicating that those who passed SCENTinel® likely did not have a SARS-CoV-2 infection. Predictability of the SCENTinel® performance was stronger when the SARS-CoV-2 Delta variant was dominant rather than when the SARS-CoV-2 Omicron variant was dominant. Additionally, SCENTinel® predicted SARS-CoV-2 positivity better than using a self-reported symptom checklist alone.

Discussion

These results indicate that SCENTinel® is a rapid assessment tool that can be used for population-level screening to monitor abrupt changes in olfactory function, and to evaluate spread of viral infections like SARS-CoV-2 that often have smell loss as a symptom.

Integrating Data Across Multiple Sites in the Northeastern United States to Examine Associations Between a Prenatal Metal Mixture and Child Cognition

We applied a novel hierarchical Bayesian weighted quantile sum (HBWQS) regression to combine data across 3 study sites to examine associations between prenatal exposure to metals and cognitive functioning in childhood. Data from 326 mother-child dyads enrolled in an ongoing cohort study, the Programming of Intergenerational Stress Mechanisms (PRISM) Study, based in New York, New York (recruitment in 2013-2020) and Boston, Massachusetts (recruitment 2011-2013), and the First Thousand Days of Life (FTDL) cohort study (recruitment 2012-2019), based in northern Virginia, were used. Arsenic, cadmium, manganese, lead, and antimony were measured in urine collected during pregnancy. Cognitive functioning was assessed in children aged 3-11 years using the National Institutes of Health Toolbox Cognition Battery. The HBWQS regression showed a negative association between the urinary metal mixture and the Cognition Early Childhood Composite Score in the PRISM New York City (β = -3.67, 95% credible interval (CrI): -7.61, -0.01) and FTDL (β = -3.76, 95% CrI: -7.66, -0.24) samples, with a similar trend in the PRISM Boston sample (β = -3.24, 95% CrI: -6.77, 0.144). We did not detect these associations in traditionally pooled models. HBWQS regression allowed us to account for site heterogeneity and detect associations between prenatal metal-mixture exposure and cognitive outcomes in childhood. Given the ubiquity of metals exposure, interventions aimed at reducing prenatal exposure may improve cognitive outcomes in children. This article is part of a Special Collection on Environmental Epidemiology.

Protocol for a construct and clinical validation study of MyCog Mobile: a remote smartphone-based cognitive screener for older adults

INTRODUCTION

Annual cognitive screening in older adults is essential for early detection of cognitive impairment, yet primary care settings face time constraints that present barriers to routine screening. A remote cognitive screener completed on a patient's personal smartphone before a visit has the potential to save primary care clinics time, encourage broader screening practices and increase early detection of cognitive decline. MyCog Mobile is a promising new remote smartphone-based cognitive screening app for primary care settings. We propose a combined construct and clinical validation study of MyCog Mobile.

METHODS AND ANALYSIS

We will recruit a total sample of 300 adult participants aged 65 years and older. A subsample of 200 healthy adult participants and a subsample of 100 adults with a cognitive impairment diagnosis (ie, dementia, mild cognitive impairment, cognitive deficits or other memory loss) will be recruited from the general population and specialty memory care centres, respectively. To evaluate the construct validity of MyCog Mobile, the healthy control sample will self-administer MyCog Mobile on study-provided smartphones and be administered a battery of gold-standard neuropsychological assessments. We will compare correlations between performance on MyCog Mobile and measures of similar and dissimilar constructs to evaluate convergent and discriminant validity. To assess clinical validity, participants in the clinical sample will self-administer MyCog Mobile on a smartphone and be administered a Mini-Cog screener and these data will be combined with the healthy control sample. We will then apply several supervised model types to determine the best predictors of cognitive impairment within the sample. Area under the receiver operating characteristic curve, accuracy, sensitivity and specificity will be the primary performance metrics for clinical validity.

ETHICS AND DISSEMINATION

The Institutional Review Board at Northwestern University (STU00214921) approved this study protocol. Results will be published in peer-reviewed journals and summaries provided to the study's funders.

Prediction of cognitive impairment using higher order item response theory and machine learning models

Timely detection of cognitive impairment (CI) is critical for the wellbeing of elderly individuals. The MyCog assessment employs two validated iPad-based measures from the NIH Toolbox® for Assessment of Neurological and Behavioral Function (NIH Toolbox). These measures assess pivotal cognitive domains: Picture Sequence Memory (PSM) for episodic memory and Dimensional Change Card Sort Test (DCCS) for cognitive flexibility. The study involved 86 patients and explored diverse machine learning models to enhance CI prediction. This encompassed traditional classifiers and neural-network-based methods. After 100 bootstrap replications, the Random Forest model stood out, delivering compelling results: precision at 0.803, recall at 0.758, accuracy at 0.902, F1 at 0.742, and specificity at 0.951. Notably, the model incorporated a composite score derived from a 2-parameter higher order item response theory (HOIRT) model that integrated DCCS and PSM assessments. The study's pivotal finding underscores the inadequacy of relying solely on a fixed composite score cutoff point. Instead, it advocates for machine learning models that incorporate HOIRT-derived scores and encompass relevant features such as age. Such an approach promises more effective predictive models for CI, thus advancing early detection and intervention among the elderly.

Remote Self-Administration of Cognitive Screeners for Older Adults Prior to a Primary Care Visit: Pilot Cross-Sectional Study of the Reliability and Usability of the MyCog Mobile Screening App

BACKGROUND

Routine cognitive screening is essential in the early detection of dementia, but time constraints in primary care settings often limit clinicians' ability to conduct screenings. MyCog Mobile is a newly developed cognitive screening system that patients can self-administer on their smartphones before a primary care visit, which can help save clinics' time, encourage broader screening practices, and increase early detection of cognitive decline.

OBJECTIVE

The goal of this pilot study was to examine the feasibility, acceptability, and initial psychometric properties of MyCog Mobile. Research questions included (1) Can older adults complete MyCog Mobile remotely without staff support? (2) Are the internal consistency and test-retest reliability of the measures acceptable? and (3) How do participants rate the user experience of MyCog Mobile?

METHODS

A sample of adults aged 65 years and older (N=51) self-administered the MyCog Mobile measures remotely on their smartphones twice within a 2- to 3-week interval. The pilot version of MyCog Mobile includes 4 activities: MyFaces measures facial memory, MySorting measures executive functioning, MySequences measures working memory, and MyPictures measures episodic memory. After their first administration, participants also completed a modified version of the Simplified System Usability Scale (S-SUS) and 2 custom survey items.

RESULTS

All participants in the sample passed the practice items and completed each measure. Findings indicate that the Mobile Toolbox assessments measure the constructs well (internal consistency 0.73 to 0.91) and are stable over an approximately 2-week delay (test-retest reliability 0.61 to 0.71). Participants' rating of the user experience (mean S-SUS score 73.17, SD 19.27) indicated that older adults found the usability of MyCog Mobile to be above average. On free-response feedback items, most participants provided positive feedback or no feedback at all, but some indicated a need for clarity in certain task instructions, concerns about participants' abilities, desire to be able to contact a support person or use in-app technical support, and desire for additional practice items.

CONCLUSIONS

Pilot evidence suggests that the MyCog Mobile cognitive screener can be reliably self-administered by older adults on their smartphones. Participants in our study generally provided positive feedback about the MyCog Mobile experience and rated the usability of the app highly. Based on participant feedback, we will conduct further usability research to improve support functionality, optimize task instructions and practice opportunities, and ensure that patients feel comfortable using MyCog Mobile. The next steps include a clinical validation study that compares MyCog Mobile to gold-standard assessments and tests the sensitivity and specificity of the measures for identifying dementia.

Life satisfaction for adolescents with developmental and behavioral disabilities during the COVID-19 pandemic

BACKGROUND

This study aimed to identify contextual factors associated with life satisfaction during the COVID-19 pandemic for adolescents with mental, emotional, behavioral, and developmental (MEBD) disabilities.

METHODS

Data were collected from a sample of 1084 adolescents aged 11-21 years from April 2020 to August 2021. This cross-sectional study used a sequential machine learning workflow, consisting of random forest regression and evolutionary tree regression, to identify subgroups of adolescents in the Environmental influences on Child Health Outcomes (ECHO) consortium who demonstrated enhanced vulnerability to lower life satisfaction as described by intersecting risk factors, protective factors, and MEBD disabilities.

RESULTS

Adolescents with a history of depression, anxiety, autism, and attention-deficit/hyperactivity disorder were particularly susceptible to decreased life satisfaction in response to unique combinations of stressors experienced during the COVID-19 pandemic. These stressors included decreased social connectedness, decreased family engagement, stress related to medical care access, pandemic-related traumatic stress, and single-caregiver households.

CONCLUSION

Findings from this study highlight the importance of interventions aimed specifically at increasing adolescent social connectedness, family engagement, and access to medical support for adolescents with MEBD disabilities, particularly in the face of stressors, such as a global pandemic.

IMPACT

Through a machine learning process, we identified contextualized risks associated with life satisfaction among adolescents with neurodevelopmental disabilities during the COVID-19 pandemic. The COVID-19 pandemic resulted in large-scale social disruptions for children and families. Such disruptions were associated with worse mental health outcomes in the general pediatric population, but few studies have examined specific subgroups who may be at heightened risk. We endeavored to close that gap in knowledge. This study highlights the importance of social connectedness, family engagement, and access to medical support as contributing factors to life satisfaction during the COVID-19 pandemic for adolescents with neurodevelopmental disabilities.

Measuring Multidimensional Aspects of Health in the Oldest Old Using the NIH Toolbox: Results From the ARMADA Study

OBJECTIVE

The percentage of older adults living into their 80s and beyond is expanding rapidly. Characterization of typical cognitive performance in this population is complicated by a dearth of normative data for the oldest old. Additionally, little attention has been paid to other aspects of health, such as motor, sensory, and emotional functioning, that may interact with cognitive changes to predict quality of life and well-being. The current study used the NIH Toolbox (NIHTB) to determine age group differences between persons aged 65-84 and 85+ with normal cognition.

METHOD

Participants were recruited in two age bands (i.e., 65-84 and 85+). All participants completed the NIHTB Cognition, Motor, Sensation, and Emotion modules. Independent-samples t-tests determined age group differences with post-hoc adjustments using Bonferroni corrections. All subtest and composite scores were then regressed on age and other demographic covariates.

RESULTS

The 65-84 group obtained significantly higher scores than the 85+ group across all cognitive measures except oral reading, all motor measures except gait speed, and all sensation measures except pain interference. Age remained a significant predictor after controlling for covariates. Age was not significantly associated with differences in emotion scores.

CONCLUSIONS

Results support the use of the NIHTB in persons over 85 with normal cognition. As expected, fluid reasoning abilities and certain motor and sensory functions decreased with age in the oldest old. Inclusion of motor and sensation batteries is warranted when studying trajectories of aging in the oldest old to allow for multidimensional characterization of health.

Seamless Integration of Computer-Adaptive Patient Reported Outcomes into an Electronic Health Record

BACKGROUND

Patient-reported outcome (PRO) measures have become an essential component of quality measurement, quality improvement, and capturing the voice of the patient in clinical care. In 2004, the National Institutes of Health endorsed the importance of PROs by initiating the Patient-Reported Outcomes Measurement Information System (PROMIS), which leverages computer-adaptive tests (CATs) to reduce patient burden while maintaining measurement precision. Historically, PROMIS CATs have been used in a large number of research studies outside the electronic health record (EHR), but growing demand for clinical use of PROs requires creative information technology solutions for integration into the EHR.

OBJECTIVES

This paper describes the introduction of PROMIS CATs into the Epic Systems EHR at a large academic medical center using a tight integration; we describe the process of creating a secure, automatic connection between the application programming interface (API) which scores and selects CAT items and Epic.

METHODS

The overarching strategy was to make CATs appear indistinguishable from conventional measures to clinical users, patients, and the EHR software itself. We implemented CATs in Epic without compromising patient data security by creating custom middleware software within the organization's existing middleware framework. This software communicated between the Assessment Center API for item selection and scoring and Epic for item presentation and results. The middleware software seamlessly administered CATs alongside fixed-length, conventional PROs while maintaining the display characteristics and functions of other Epic measures, including automatic display of PROMIS scores in the patient's chart. Pilot implementation revealed differing workflows for clinicians using the software.

RESULTS

The middleware software was adopted in 27 clinics across the hospital system. In the first 2 years of hospital-wide implementation, 793 providers collected 70,446 PROs from patients using this system.

CONCLUSION

This project demonstrated the importance of regular communication across interdisciplinary teams in the design and development of clinical software. It also demonstrated that implementation relies on buy-in from clinical partners as they integrate new tools into their existing clinical workflow.

Primary care detection of cognitive impairment leveraging health and consumer technologies in underserved US communities: protocol for a pragmatic randomised controlled trial of the MyCog paradigm

INTRODUCTION

Early identification of cognitive impairment (CI), including Alzheimer's disease and related dementias (ADRD), is a top public health priority. Yet, CI/ADRD is often undetected and underdiagnosed within primary care settings, and in health disparate populations. The MyCog paradigm is an iPad-based, self-administered, validated cognitive assessment based on the National Institutes of Health (NIH) Toolbox Cognition Battery and coupled with clinician decision-support tools that is specifically tailored for CI/ADRD detection within diverse, primary care settings.

METHODS AND ANALYSIS

We will conduct a two-arm, primary care practice-randomised (N=24 practices; 45 257 active patients at the proposed practices), pragmatic trial among geographically diverse Oak Street Health sites to test the effectiveness of the MyCog paradigm to improve early detection CI/ADRD among low socioeconomic, black and Hispanic older adults compared with usual care. Participating practices randomised to the intervention arm will impart the MyCog paradigm as a new standard of care over a 3-year implementation period; as the cognitive component for Annual Wellness Visits and for any patient/informant-reported or healthcare provider-suspected cognitive concern. Rates of detected (cognitive test suggesting impairment) and/or diagnosed (relevant International Classification of Diseases-9/10 [ICD-9/10] code) cognitive deficits, impairments or dementias including ADRD will be our primary outcome of study compared between arms. Secondary outcomes will include ADRD severity (ie, mild or later stage), rates of cognitive-related referrals and rates of family member or caregiver involvement in ADRD care planning. We will use generalised linear mixed models to account for clustered study design. Secondary models will adjust for subject, clinic or visit-specific characteristics. We will use mixed-methods approaches to examine fidelity and cost-effectiveness of the MyCog paradigm.

ETHICS AND DISSEMINATION

The Institutional Review Board at Advarra has approved the study protocol (Pro00064339). Results will be published in peer-reviewed journals and summaries will be provided to the funders of the study.

TRIAL REGISTRATION NUMBER

NCT05607732.

The Paradox in Positive and Negative Aspects of Emotional Functioning Among Older Adults with Early Stages of Cognitive Impairment

Introduction: Emotional functioning in older adults is influenced by normal aging and cognitive impairment, likely heterogeneous across positive versus negative aspects of emotional functioning. Little is known about positive emotional experiences at the early stages of cognitive impairment. Methods: We assessed different aspects of emotional functioning among 448 participants aged 65+ (Normal Control (NC) = 276, Mild Cognitive Impairment (MCI) = 103, and mild dementia of the Alzheimer type (mild DAT) = 69) and tested moderators. Results: Compared to NC, older adults with MCI and mild DAT have maintained many positive aspects of emotional functioning, despite higher levels of negative affect, sadness, and loneliness. Among the oldest-old, the mild DAT group experienced higher fear and lower self-efficacy. Discussion: Older adults at early stages of cognitive impairment can experience positive aspects of emotional functioning, such as positive affect, purpose, and life satisfaction, all of which are important buildable psychological resources for coping.

Cognitive function, self-management, and outcomes among liver transplant recipients: LivCog, a multicenter, prospective study

Liver transplantation is a life-saving option for decompensated cirrhosis. Liver transplant recipients require advanced self-management skills, intact cognitive skills, and care partner support to improve long-term outcomes. Gaps remain in understanding post-liver transplant cognitive and health trajectories, and patient factors such as self-management skills, care partner support, and sleep. Our aims are to (1) assess pre-liver transplant to post-liver transplant cognitive trajectories and identify risk factors for persistent cognitive impairment; (2) evaluate associations between cognitive function and self-management skills, health behaviors, functional health status, and post-transplant outcomes; and (3) investigate potential mediators and moderators of associations between cognitive function and post-liver transplant outcomes. LivCog is a longitudinal, prospective observational study that will enroll 450 adult liver transplant recipients and their caregivers/care partners. The duration of the study is 5 years with 24 additional months of patient follow-up. Data will be collected from participants at 1, 3, 12, and 24 months post-transplant. Limited pre-liver transplant data will also be collected from waitlisted candidates. Data collection methods include interviews, surveys, cognitive assessments, and actigraphy/sleep diary measures. Patient measurements include sociodemographic characteristics, pretransplant health status, cognitive function, physical function, perioperative measures, medical history, transplant history, self-management skills, patient-reported outcomes, health behaviors, and clinical outcomes. Caregiver measures assess sociodemographic variables, health literacy, health care navigation skills, self-efficacy, care partner preparedness, nature and intensity of care, care partner burden, and community participation. By elucidating various health trajectories from pre-liver transplant to 2 years post-liver transplant, LivCog will be able to better characterize recipients at higher risk of cognitive impairment and compromised self-management. Findings will inform interventions targeting health behaviors, self-management, and caregiver supports to optimize outcomes.

Validation of a tablet-based assessment of auditory sensitivity for researchers

Quantifying hearing acuity is increasingly important across a wide range of research areas in the behavioral and neurosciences. Scientists have relied on either self-reported hearing status or the availability of diagnostic hearing assessment in past studies. There remains a need for a valid and reliable assessment of auditory sensitivity that can provide estimates of the magnitude of hearing loss, if present, without requirements for professional audiologists, facilities, and equipment that are needed to conduct a diagnostic hearing assessment. The goal of this experiment was to validate the NIH Toolbox® Hearing Threshold Test (HTT), a tablet-based hearing assessment available via iPad application that uses consumer-grade headphones, on a clinical sample of children and adults with varying degrees of hearing acuity. Electroacoustic analysis of the hearing assessment application and headphones demonstrated acoustic outputs within established conformity standards for hearing assessment. Twenty-seven children and 63 adults participated in a standard diagnostic hearing assessment and the experimental tablet-based assessment. The results showed that thresholds from the tablet-based assessment were highly correlated with thresholds from the clinical hearing assessment (r = .83-.93) for children and adults for all frequencies and across a range of levels of hearing acuity. The HTT also met clinical test-retest reliability standards (Cronbach's α > .86). The tablet-based hearing assessment provides acceptable estimates of hearing levels for children and adults when diagnostic audiometric assessment capabilities are not available.

Baseline characterization of the ARMADA (Assessing Reliable Measurement in Alzheimer's Disease) study cohorts

INTRODUCTION

The National Institutes of Health (NIH) Toolbox (NIHTB) provides computerized measures of cognition, emotion, sensation, and motor abilities across the lifespan. The ARMADA (Assessing Reliable Measurement in Alzheimer's Disease and Cognitive Aging) study validated the NIHTB in individuals across the cognitive aging spectrum. This article reports the characteristics of our sample of participants.

METHODS

Participants were recruited across nine sites and classified clinically as cognitively normal (NC), with mild cognitive impairment (MCI), or with dementia of the Alzheimer's type (DAT.) They completed the NIHTB at multiple time points and many had at least one Alzheimer's biomarker previously obtained.

RESULTS

Groups differed with respect to dementia severity levels, as anticipated, but were well-matched across many demographic characteristics.

DISCUSSION

The ARMADA study demographics and baseline characteristics provide a suitable sample for validating the NIHTB across the cognitive aging spectrum. Other enriched samples (African American participants, Spanish NIHTB, 85+ years of age) will be reported elsewhere.

HIGHLIGHTS

There is a need for assessments that can detect the early stages of cognitive decline in older adults. The ARMADA (Assessing Reliable Measurement in Alzheimer's Disease and Cognitive Aging) study will validate the National Institutes of Health (NIH) Toolbox across the aging spectrum, including mild cognitive impairment (MCI) and dementia of the Alzheimer's type (DAT). Here we report the characteristics of participants. Groups were well-matched across most demographic characteristics, and clinical characteristics differed as expected. ARMADA study cohorts reflect their respective clinical syndromes for validating the NIH Toolbox.

Perspective: Advancing dietary guidance for cognitive health - focus on solutions to harmonization of test selection, implementation, and evaluation

This perspective paper is a product of a workshop of experts convened by Institute for the Advancement of Food and Nutrition Sciences (IAFNS), a non-profit organization that brings together scientists from government, academia, and industry to catalyze science relevant to food and nutrition. An expert group was convened in March 2022 to discuss the current issues surrounding cognitive task selection in nutrition research, with a focus on solutions towards informing dietary guidance for cognitive health, to address a gap identified in the 2020 US DGAC report, specifically the "considerable variation in testing methods used, [and] inconsistent validity and reliability of cognitive testing methods" (p.31). To address this issue, we firstly undertook an umbrella review of relevant reviews already undertaken; these indicate agreement on some of the issues that affect heterogeneity in task selection, and on many of the fundamental principles underlying selection of cognitive outcome measures. However, resolving the points of disagreement is critical to ensuring meaningful impact on the issue of heterogeneity in task selection; these issues hamper evaluation of existing data for informing dietary guidance. This summary of the literature is therefore followed by the expert group's perspective in the form of a discussion of potential solutions to these challenges, with the aim of building on the work of previous reviews in the area and advancing dietary guidance for cognitive health. STATEMENT OF SIGNIFICANCE: Despite several high-quality reviews in this field over the last two decades, there has been little in the way of substantive change in the methods being used to conduct studies, hampering harmonization of the evidence and thus, its utility for informing dietary guidance. The present paper comprehensively updates the field by firstly providing an umbrella review of the published reviews, followed by the IAFNS expert group's perspective on how to move the field forward by addressing the challenges and areas of disagreement in the existing reviews. REGISTERED ON PROSPERO: CRD42022348106. Data described in the manuscript, code book, and analytic code will be made publicly and freely available without restriction at doi.org/10.17605/OSF.IO/XRZCK.

Prenatal exposure to PM2.5 and childhood cognition: Accounting for between-site heterogeneity in a pooled analysis of ECHO cohorts in the Northeastern United States

BACKGROUND

Emerging studies have investigated the adverse health effects of PM_2.5 using data from multiple cohorts, and results often are not generalizable across cohorts. We aimed to assess associations between prenatal PM_2.5 and childhood cognition in two U.S. cohorts while accounting for between-site heterogeneity.

METHODS

Analyses included 348 mother-child dyads enrolled in the dual site (New York City and Boston) PRogramming of Intergenerational Stress Mechanisms (PRISM) cohort and in the First Thousand Days of Life (FTDL) study (Northern Virginia) participating in the Environmental influences on Child Health Outcomes (ECHO) national consortium. Residential prenatal PM_2.5 exposure was estimated using a validated satellite-based model and childhood cognition was measured using the NIH Toolbox Cognition Battery at three to eight years of age. We used a log-linear model applied to contingency tables formed by cross-classifying covariates by site to examine between-site heterogeneity using 3^rd trimester PM_2.5 exposure, age-corrected cognition scores, and covariates potentially causing heterogeneities. Multivariable linear regression models informed by the combinability analysis were used to estimate the coefficients and 95% confidence intervals (CIs) for the association between 3^rd trimester PM_2.5 exposure and age-corrected cognition scores (mean = 100, SD = 15).

RESULTS

The log-linear model indicated that inter-study associations were similar between PRISM-NYC and FTDL, which were different from those in PRISM-Boston. Accordingly, we combined the data of PRISM-NYC and FTDL cohorts. We observed associations between 3^rd trimester PM_2.5 and cognition scores, findings were varying by site, childsex, and test. For example, a 1 μg/m³ increase of 3^rd trimester PM_2.5 was associated with -4.35 (95% CI = -8.73, -0.25) mean early childhood cognition scores in females in PRISM-Boston. In the pooled NYC + FTDL site, the association between PM_2.5 and childhood cognition may be modified by maternal education and urbanicity.

CONCLUSIONS

We found associations between prenatal PM_2.5 and impaired childhood cognition. Since multi-site analyses are increasingly conducted, our findings suggest the needed awareness of between-site heterogeneity.

ARMADA: Assessing reliable measurement in Alzheimer's disease and cognitive aging project methods

INTRODUCTION

Early detection of cognitive decline in older adults is a public health priority. Advancing Reliable Measurement in Alzheimer's Disease and Cognitive Aging (ARMADA), a multisite study, is validating cognition, emotion, motor, and sensory modules of the National Institutes of Health Toolbox for Assessment of Neurological and Behavioral Function (NIHTB) in the aging spectrum from cognitively normal to dementia of the Alzheimer's type (DAT).

METHODS

Participants 65 to 85 years old, in demographic groups racially proportional to the general US population, are recruited in one of three groups to validate the NIHTB: cognitively normal, amnestic mild cognitive impairment (aMCI), or mild DAT. Additional special emphasis cohorts include (1) Blacks in the three clinical groups; (2) Spanish-speakers in the three clinical groups; (3) cognitively normal, population-proportional, over age 85.

DISCUSSION

Longitudinal study will determine whether NIHTB can predict cognitive decline and is associated with Alzheimer's disease biomarkers. Here, we detail the methods for the ARMADA study.

Youth Well-being During the COVID-19 Pandemic

OBJECTIVES

The family stress model proposes economic hardship results in caregiver distress and relational problems, which negatively impact youth outcomes. We extend this model to evaluate the impact of coronavirus disease 2019 pandemic-related family hardships on caregiver and youth stress, and, in turn, youth's psychological well-being. We also investigate how social supports moderate this relationship.

METHODS

We used 2 samples of cross-sectional survey data collected between May 2020 and May 2021: children aged 2 to 12 years (n = 977) and adolescents aged 11 to 17 years (n = 669). Variables included pandemic-related family hardships, stress, social support, and youth life satisfaction. Data were analyzed using structural equation modeling.

RESULTS

Experiencing more pandemic-related family hardships was associated with increased caregiver and youth stress (b = 0.04 to 0.21, SE = 0.01-0.02) and, in turn, decreased youth life satisfaction (b = -0.36 to -0.38, SE = 0.04-0.07). Social connectedness (b^ = 0.11-0.17, SE = 0.04) and family engagement (b^ = 0.12-0.18, SE = 0.05-0.06) had direct positive associations with life satisfaction; for children aged 2 to 12 years, greater family engagement was associated with decreased effect of child stress on life satisfaction (b^ = 0.15, SE = 0.05). For adolescents, females had higher levels of stress compared with males (b^ = 0.40, SE = 0.6), and having anxiety and/or depression was associated with decreased life satisfaction (b^ = -0.24, SE = 0.11).

CONCLUSIONS

Caregivers and youth who experienced more coronavirus disease 2019 pandemic hardships had higher levels of stress, particularly adolescent females. Although stress negatively impacted life satisfaction across all ages, family engagement was a protective factor for children aged 2 to 12 years, whereas having anxiety and/or depression was a risk factor for adolescents. For all youth, however, being more socially connected and engaged with family promoted life satisfaction.

Lessons from Detecting Cognitive Impairment Including Dementia (DetectCID) in Primary Care

Background: Cognitive impairment, including dementia, is frequently under-detected in primary care. The Consortium for Detecting Cognitive Impairment, including Dementia (DetectCID) convenes three multidisciplinary teams that are testing novel paradigms to improve the frequency and quality of patient evaluations for detecting cognitive impairment in primary care and appropriate follow-up. Objective: Our objective was to characterize the three paradigms, including similarities and differences, and to identify common key lessons from implementation. Methods: A qualitative evaluation study with dementia specialists who were implementing the detection paradigms. Data was analyzed using content analysis. Results: We identified core components of each paradigm. Key lessons emphasized the importance of engaging primary care teams, enabling primary care providers to diagnose cognitive disorders and provide ongoing care support, integrating with the electronic health record, and ensuring that paradigms address the needs of diverse populations. Conclusion: Approaches are needed that address the arc of care from identifying a concern to post-diagnostic management, are efficient and adaptable to primary care workflows, and address a diverse aging population. Our work highlights approaches to partnering with primary care that could be useful across specialties and paves the way for developing future paradigms that improve differential diagnosis of symptomatic cognitive impairment, identifying not only its presence but also its specific syndrome or etiology.

Developing a common metric for depression across adulthood: Linking PROMIS depression with the Edinburgh Postnatal Depression Scale

Depression is a leading mental health concern across the U.S. and worldwide. There are myriad assessments to evaluate depressive symptoms, including the Edinburgh Postnatal Depression Scale (EPDS), which is widely used to evaluate women's pre- and postnatal depression but not as prevalent at other timepoints in adulthood, limiting its utility for longitudinal research. As part of the National Institutes of Health's (NIH) Environmental influences on Child Health Outcomes (ECHO) Research Program, the current study sought to develop a common metric so that scores on the EPDS can be converted to the standardized Patient-Reported Measurement Information System (PROMIS®) T-score metric. Drawing on data from the ECHO-Prenatal Alcohol in SIDS and Stillbirth cohort, this study used a single-group linking design, where 1,263 mothers completed the EPDS and PROMIS-Depression measures at the same time. Score linking was conducted using equipercentile and item response theory (IRT) methods. Results showed both linking methods provide robust, congruent results, and subgroup invariance held across age, race, ethnicity, education, and geographic location. The IRT-based unidimensional fixed-parameter calibration was selected due to its model simplicity, and a crosswalk table was established to convert scores from the EPDS to PROMIS T-scores. Overall, this study provides a way to aggregate data across various depression measures and timepoints, such that researchers and clinicians now have the ability to directly compare and combine EPDS data with PROMIS and other depression measures already score-linked to PROMIS. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Preliminary Results of MyCog, a Brief Assessment for the Detection of Cognitive Impairment in Primary Care

Early detection of Cognitive impairment (CI) is imperative to identify potentially treatable underlying conditions or provide supportive services when due to progressive conditions such as Alzheimer’s Disease. While primary care settings are ideal for identifying CI, it frequently goes undetected. We developed ‘MyCog’, a brief technology-enabled, 2-step assessment to detect CI and dementia in primary care settings. We piloted MyCog in 80 participants 65 and older recruited from an ongoing cognitive aging study. Cases were identified either by a documented diagnosis of dementia or mild cognitive impairment (MCI) or based on a comprehensive cognitive battery. Administered via an iPad, Step 1 consists of a single self-report item indicating concern about memory or other thinking problems and Step 2 includes two cognitive assessments from the NIH Toolbox: Picture Sequence Memory (PSM) and Dimensional Change Card Sorting (DCCS). 39%(31/80) participants were considered cognitively impaired. Those who expressed concern in Step 1 (n=52, 66%) resulted in a 37% false positive and 3% false negative rate. With the addition of the PSM and DCCS assessments in Step 2, the paradigm demonstrated 91% sensitivity, 75% specificity and an area under the ROC curve (AUC)=0.82. Steps 1 and 2 had an average administration time of <7 minutes. We continue to optimize MyCog by 1) examining additional items for Step 1 to reduce the false positive rate and 2) creating a self-administered version to optimize use in clinical settings. With further validation, MyCog offers a practical, scalable paradigm for the routine detection of cognitive impairment and dementia.

FEASIBILITY AND ACCEPTABILITY OF DETECTING COGNITIVE IMPAIRMENT AND DEMENTIA IN PRIMARY CARE PRACTICES

As the age of the US population increases, so does cognitive impairment (CI); therefore early detection of CI is critical for ensuring its appropriate management. As part of a NINDS Consortium to detect CI and dementia in primary care (DetectCID), we are implementing and evaluating a brief 2-step CI detection paradigm (MyCog), that can be delivered in clinics with diverse populations via the electronic health record (step 1) and iPad (step 2). We conducted focus groups with 25 clinicians and administrative leaders from academic and community primary care practices to 1) understand how CI is being assessed, and 2) evaluate the feasibility of implementing the MyCog paradigm into existing primary care workflows. Several key themes emerged from the discussions. No proactive detection strategy for CI was regularly used outside of the Medicare Annual Wellness Visits (AWV); variable assessments including the Minicog, MoCA, or MMSE were used to fulfill the AWV requirement. Regarding the feasibility of our MyCog Paradigm, our 2-step process was positively received, with the brief case-finding step 1 satisfying AWV requirements and replacing the longer assessments currently being used. Clinicians preferred that step 2 be self-administered due to limited clinician time for wellness visits, and highlighted logistical challenges such as room availability and storage and maintenance of the iPad. Overall, clinicians felt that the identification of CI was valuable and supported standardization, but indicated regular case finding was unlikely without clear guidance on clinical decision-making.

Reliability and Validity of the Spanish Language Measures of the NIH Toolbox Cognition Battery

Objective

The NIH Toolbox for Assessment of Neurological and Behavioral Function (NIHTB) was developed under contract from the National Institutes of Health to create a set of easy-to-administer neuropsychological measures, for use across the lifespan (ages 3-85). The NIHTB Cognition Battery (NIHTB-CB) includes two language measures that were developed, calibrated, and normed separately in English and Spanish. This analysis presents the test-retest reliability and construct validity of the Spanish-language picture vocabulary test (S-PVT) and oral reading recognition test (S-ORRT) among adults.

Participants and Method

Participants were adults age 18-85 who took part in the NIHTB norming study in Spanish (N = 408, Age: M = 44.1, SD = 16.7; Education: M = 10.7, SD = 4.3; 65.0% female). Of these, 48 repeated the battery 1 week later. Both the S-PVT and the S-ORRT were administered using computer adaptive testing and scored using item response theory. Spearman’s correlations were used to evaluate test-retest reliability. Convergent validity was evaluated by correlating S-PVT scores with scores on the Batería-III Woodcock-Muñoz Vocabulario Sobre Dibujos, and by correlating S-ORRT scores with scores on a 48-item version of the Word Accentuation Test. Adjusted Spearman’s correlations and general linear models related scores to age, education, and sex.

Results

Both the S-PVT (ρ = 0.87, p &lt; .001) and the S-ORRT (ρ = 0.88, p &lt; .001) demonstrated good test-retest reliability. Good convergent validity was found for both the S-PVT (ρ = 0.76, p &lt; .001) and the S-ORRT (ρ = 0.65, p &lt; .001). Scores on the S-PVT were positively related to education (ρ = 0.38, p &lt; .001), and scores on the S-ORRT were negatively related to age (ρ = -0.18, p &lt; .01) and positively related to education (ρ = 0.30, p &lt; .001).

Conclusions

The Spanish language measures of the NIHTB-CB demonstrated acceptable reliability and validity, suggesting they can be used to measure language ability among Spanish-speaking adults in the United States.

Nih toolbox premorbid ability adjustments: Application in a traumatic brain injury sample

PURPOSE/OBJECTIVE

Metrics to estimate premorbid cognitive ability, such as word reading tests, are important for clinical determination of cognitive changes following brain injury. In the present study, reading adjusted scores for the National Institutes of Health Toolbox Cognition Battery (NIHTB-CB) fluid tests were developed and validated with a sample of individuals with traumatic brain injury (TBI), to evaluate the clinical utility of reading-adjusted scores. Research Method/Design: The development sample included 843 adult participants, ages 20-85, from the NIHTB-CB standardization sample. A sample of 158 participants with complicated mild or moderate TBI (n = 74) or severe TBI (n = 84) were administered the NIHTB-CB, and comprised the validation sample. Scores were derived for the five fluid tests using four adjustment models: age-only, demographic-only, age-and-reading, and demographic-and-reading referenced scores.

RESULTS

Estimated premorbid ability varies depending on the reference model. Scores from each of the four reference models differentiated the comparison and TBI samples at the group level. However, performance varied by premorbid ability.

CONCLUSIONS/IMPLICATIONS

Premorbid ability affects identification of cognitive difficulties after TBI. Reading referenced scores provide an individualized estimate of the effects of premorbid ability than demographic characteristics alone. Each model identified a similar number of individuals as having cognitive difficulties; however, the models differed on which individuals had cognitive difficulties. The models had higher disagreement rates in the clinical compared with the comparison sample, particularly for individuals with lower premorbid ability. Clinical use and caveats are discussed. (PsycINFO Database Record

Interpreting Patterns of Low Scores on the NIH Toolbox Cognition Battery

INTRODUCTION

The National Institutes of Health Toolbox for Assessment of Neurological and Behavioral Function Cognition Battery is comprised of seven cognitive tests, including two tests measuring crystallized cognitive ability (i.e., vocabulary and reading) and five tests measuring fluid cognitive functioning (i.e., working memory, memory, speed of processing, and executive functioning). This study presents comprehensive base rate tables for the frequency of low scores in adults and older adults from the normative sample.

METHODS

Participants were 843 adults, ages 20-85, from the NIH Toolbox standardization sample who completed all seven cognition tests. Rates of low scores were derived for standard age-adjusted and fully-demographically-adjusted scores at multiple cut-scores. Base rates were stratified by education, crystallized intellectual ability, and cognitive domain.

RESULTS

Using the five demographically-adjusted fluid cognitive test scores, 45.9% of adults obtained one or more scores at or below the 16th percentile, and 16.8% obtained one or more score at or below the 5th percentile, which is consistent with findings from other neurocognitive test batteries.

DISCUSSION

Based on the study findings, nearly 50% of adults in the general population would meet psychometric criteria for a diagnosis of the Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (DSM-5) Mild Neurocognitive Disorder (MND). We developed new psychometric criteria for identifying MND using the NIH Toolbox Cognition Battery that reduce the false positive rate. Knowing these multivariate normative base rates will help researchers and clinicians interpret NIH Toolbox scores in people with neurodevelopmental, psychiatric, medical, neurological, and neurodegenerative disorders that affect cognitive functioning.

Validation of the NIH Toolbox in Individuals with Neurologic Disorders

OBJECTIVE

Individuals with spinal cord injury (SCI), traumatic brain injury (TBI), and stroke experience a variety of neurologically related deficits across multiple domains of function. The NIH Toolbox for the Assessment of Neurological and Behavioral Function (NIHTB) examines motor, sensation, cognition, and emotional functioning. The purpose of this paper is to establish the validity of the NIHTB in individuals with neurologic conditions.

METHODS

Community-dwelling individuals with SCI (n = 209), TBI (n = 184), or stroke (n = 211) completed the NIHTB. Relative risks for impaired performance were examined relative to a matched control groups.

RESULTS

The largest group differences were observed on the Motor domain and for the Fluid Cognition measures. All groups were at increased risk for motor impairment relative to normative standards and matched controls. Fluid cognitive abilities varied across groups such that individuals with stroke and TBI performed more poorly than individuals with SCI; increased relative risks for impaired fluid cognition were seen for individuals in the stroke and TBI groups, but not for those in the SCI group. All three neurologic groups performed normally on most measures in the Sensation Battery, although TBI participants evidenced increased risk for impaired odor identification and the stroke group showed more vision difficulties. On the Emotion Battery, participants in all three groups showed comparably poor psychological well-being, social satisfaction, and self-efficacy, whereas the TBI group also evidenced slightly increased negative affect.

CONCLUSIONS

Data provide support for the validity of the NIHTB in individuals with neurologic conditions.

Hand-related physical function in rheumatic hand conditions: a protocol for developing a patient-reported outcome measurement instrument

INTRODUCTION

There is no consensus about what constitutes the most appropriate patient-reported outcome measurement (PROM) instrument for measuring physical function in patients with rheumatic hand conditions. Existing instruments lack psychometric testing and vary in feasibility and their psychometric qualities. We aim to develop a PROM instrument to assess hand-related physical function in rheumatic hand conditions.

METHODS AND ANALYSIS

We will perform a systematic search to identify existing PROMs to rheumatic hand conditions, and select items relevant for hand-related physical function as well as those items from the Patient Reported Outcomes Measurement Information System (PROMIS) Physical Function (PF) item bank that are relevant to patients with rheumatic hand conditions. Selection will be based on consensus among reviewers. Content validity of selected items will be established through the use of focus groups. If patients deem necessary, we will develop new items based on the patients' input. We will examine whether it is valid to score all selected and developed items on the same scale as the original items from the PROMIS PF item bank. Our analyses will follow the methods used for calibrating the original PROMIS PF item bank in US samples, which were largely based on the general PROMIS approach.

ETHICS AND DISSEMINATION

This study will be carried out in accordance with the Helsinki Declaration. Ethics approvals will be obtained where necessary, and signed informed consent will be obtained from all participants. We aim to disseminate the results of the study through publication in international peer-reviewed journals and at international conferences.

Overview of the Spinal Cord Injury--Quality of Life (SCI-QOL) measurement system

CONTEXT/OBJECTIVE

The Spinal Cord Injury--Quality of Life (SCI-QOL) measurement system was developed to address the shortage of relevant and psychometrically sound patient reported outcome (PRO) measures available for clinical care and research in spinal cord injury (SCI) rehabilitation. Using a computer adaptive testing (CAT) approach, the SCI-QOL builds on the Patient Reported Outcomes Measurement Information System (PROMIS) and the Quality of Life in Neurological Disorders (Neuro-QOL) initiative. This initial manuscript introduces the background and development of the SCI-QOL measurement system. Greater detail is presented in the additional manuscripts of this special issue.

DESIGN

Classical and contemporary test development methodologies were employed. Qualitative input was obtained from individuals with SCI and clinicians through interviews, focus groups, and cognitive debriefing. Item pools were field tested in a multi-site sample (n=877) and calibrated using item response theory methods. Initial reliability and validity testing was performed in a new sample of individuals with traumatic SCI (n=245).

SETTING

Five Model SCI System centers and one Department of Veterans Affairs Medical Center across the United States.

PARTICIPANTS

Adults with traumatic SCI.

INTERVENTIONS

n/a

OUTCOME MEASURES

n/a

RESULTS

The SCI-QOL consists of 19 item banks, including the SCI-Functional Index banks, and 3 fixed-length scales measuring physical, emotional, and social aspects of health-related QOL (HRQOL).

CONCLUSION

The SCI-QOL measurement system consists of psychometrically sound measures for individuals with SCI. The manuscripts in this special issue provide evidence of the reliability and initial validity of this measurement system. The SCI-QOL also links to other measures designed for a general medical population.

Methodology for the development and calibration of the SCI-QOL item banks

OBJECTIVE

To develop a comprehensive, psychometrically sound, and conceptually grounded patient reported outcomes (PRO) measurement system for individuals with spinal cord injury (SCI).

METHODS

Individual interviews (n=44) and focus groups (n=65 individuals with SCI and n=42 SCI clinicians) were used to select key domains for inclusion and to develop PRO items. Verbatim items from other cutting-edge measurement systems (i.e. PROMIS, Neuro-QOL) were included to facilitate linkage and cross-population comparison. Items were field tested in a large sample of individuals with traumatic SCI (n=877). Dimensionality was assessed with confirmatory factor analysis. Local item dependence and differential item functioning were assessed, and items were calibrated using the item response theory (IRT) graded response model. Finally, computer adaptive tests (CATs) and short forms were administered in a new sample (n=245) to assess test-retest reliability and stability.

PARTICIPANTS AND PROCEDURES

A calibration sample of 877 individuals with traumatic SCI across five SCI Model Systems sites and one Department of Veterans Affairs medical center completed SCI-QOL items in interview format.

RESULTS

We developed 14 unidimensional calibrated item banks and 3 calibrated scales across physical, emotional, and social health domains. When combined with the five Spinal Cord Injury--Functional Index physical function banks, the final SCI-QOL system consists of 22 IRT-calibrated item banks/scales. Item banks may be administered as CATs or short forms. Scales may be administered in a fixed-length format only.

CONCLUSIONS

The SCI-QOL measurement system provides SCI researchers and clinicians with a comprehensive, relevant and psychometrically robust system for measurement of physical-medical, physical-functional, emotional, and social outcomes. All SCI-QOL instruments are freely available on Assessment CenterSM.

Bringing PROMIS to practice: brief and precise symptom screening in ambulatory cancer care

BACKGROUND

Supportive oncology practice can be enhanced by the integration of a brief and validated electronic patient-reported outcome assessment into the electronic health record (EHR) and clinical workflow.

METHODS

Six hundred thirty-six women receiving gynecologic oncology outpatient care received instructions to complete clinical assessments through Epic MyChart, an EHR patient communication portal. Patient Reported Outcomes Measurement Information System (PROMIS) computer adaptive tests (CATs) were administered to assess fatigue, pain interference, physical function, depression, and anxiety. Checklists identified psychosocial concerns, informational and nutritional needs, and risk factors for inadequate nutrition. Assessment results, including PROMIS T scores with documented severity thresholds, were immediately populated in the EHR. Clinicians were notified of clinically elevated symptoms through EHR messages. EHR integration was designed to provide automated triage to social work providers for psychosocial concerns, to health educators for information, and to dietitians for nutrition-related concerns.

RESULTS

Four thousand forty-two MyChart messages sent, and 3203 (79%) were reviewed by patients. The assessment was started by 1493 patients (37%), and once they started, 93% (1386 patients) completed the assessment. According to first assessments only, 49.8% of the patients who reviewed the MyChart message completed the assessment. Mean PROMIS CAT T scores indicated a lower level of physical function and elevated anxiety in comparison with the general population. Fatigue, pain, and depression scores were comparable to those of the general population. Impaired physical functioning was the most common basis for clinical alerts and occurred in 4% of the patients.

CONCLUSIONS

PROMIS CATs were used to measure common cancer symptoms in routine oncology outpatient care. Immediate EHR integration facilitated the use of symptom reporting as the basis for referral to psychosocial and supportive care.

Reliability and validity of composite scores from the NIH Toolbox Cognition Battery in adults

This study describes psychometric properties of the NIH Toolbox Cognition Battery (NIHTB-CB) Composite Scores in an adult sample. The NIHTB-CB was designed for use in epidemiologic studies and clinical trials for ages 3 to 85. A total of 268 self-described healthy adults were recruited at four university-based sites, using stratified sampling guidelines to target demographic variability for age (20-85 years), gender, education, and ethnicity. The NIHTB-CB contains seven computer-based instruments assessing five cognitive sub-domains: Language, Executive Function, Episodic Memory, Processing Speed, and Working Memory. Participants completed the NIHTB-CB, corresponding gold standard validation measures selected to tap the same cognitive abilities, and sociodemographic questionnaires. Three Composite Scores were derived for both the NIHTB-CB and gold standard batteries: "Crystallized Cognition Composite," "Fluid Cognition Composite," and "Total Cognition Composite" scores. NIHTB Composite Scores showed acceptable internal consistency (Cronbach's alphas=0.84 Crystallized, 0.83 Fluid, 0.77 Total), excellent test-retest reliability (r: 0.86-0.92), strong convergent (r: 0.78-0.90) and discriminant (r: 0.19-0.39) validities versus gold standard composites, and expected age effects (r=0.18 crystallized, r=-0.68 fluid, r=-0.26 total). Significant relationships with self-reported prior school difficulties and current health status, employment, and presence of a disability provided evidence of external validity. The NIH Toolbox Cognition Battery Composite Scores have excellent reliability and validity, suggesting they can be used effectively in epidemiologic and clinical studies.

NIH Toolbox Cognition Battery (CB): validation of executive function measures in adults

This study describes psychometric properties of the NIH Toolbox Cognition Battery (NIHTB-CB) executive function measures in an adult sample. The NIHTB-CB was designed for use in epidemiologic studies and clinical trials for ages 3 to 85. A total of 268 self-described healthy adults were recruited at four university-based sites, using stratified sampling guidelines to target demographic variability for age (20-85 years), gender, education and ethnicity. The NIHTB-CB contains two computer-based instruments assessing executive function: the Dimensional Change Card Sort (a measure of cognitive flexibility) and a flanker task (a measure of inhibitory control and selective attention). Participants completed the NIHTB-CB, corresponding gold standard convergent and discriminant measures, and sociodemographic questionnaires. A subset of participants (N=89) was retested 7 to 21 days later. Results reveal excellent sensitivity to age-related changes during adulthood, excellent test-retest reliability, and adequate to good convergent and discriminant validity. The NIH Toolbox EF measures can be used effectively in epidemiologic and clinical studies.

NIH Toolbox Cognition Battery (NIHTB-CB): list sorting test to measure working memory

The List Sorting Working Memory Test was designed to assess working memory (WM) as part of the NIH Toolbox Cognition Battery. List Sorting is a sequencing task requiring children and adults to sort and sequence stimuli that are presented visually and auditorily. Validation data are presented for 268 participants ages 20 to 85 years. A subset of participants (N=89) was retested 7 to 21 days later. As expected, the List Sorting Test had moderately high correlations with other measures of working memory and executive functioning (convergent validity) but a low correlation with a test of receptive vocabulary (discriminant validity). Furthermore, List Sorting demonstrates expected changes over the age span and has excellent test-retest reliability. Collectively, these results provide initial support for the construct validity of the List Sorting Working Memory Measure as a measure of working memory. However, the relationship between the List Sorting Test and general executive function has yet to be determined.

Development of a vision-targeted health-related quality of life item measure

PURPOSE

To develop a vision-targeted health-related quality of life (HRQOL) measure for the NIH Toolbox for the Assessment of Neurological and Behavioral Function.

METHODS

We conducted a review of existing vision-targeted HRQOL surveys and identified color vision, low luminance vision, distance vision, general vision, near vision, ocular symptoms, psychosocial well-being, and role performance domains. Items in existing survey instruments were sorted into these domains. We selected non-redundant items and revised them to improve clarity and to limit the number of different response options. We conducted 10 cognitive interviews to evaluate the items. Finally, we revised the items and administered them to 819 individuals to calibrate the items and estimate the measure's reliability and validity.

RESULTS

The field test provided support for the 53-item vision-targeted HRQOL measure encompassing 6 domains: color vision, distance vision, near vision, ocular symptoms, psychosocial well-being, and role performance. The domain scores had high levels of reliability (coefficient alphas ranged from 0.848 to 0.940). Validity was supported by high correlations between National Eye Institute Visual Function Questionnaire scales and the new-vision-targeted scales (highest values were 0.771 between psychosocial well-being and mental health, and 0.729 between role performance and role difficulties), and by lower mean scores in those groups self-reporting eye disease (F statistic with p < 0.01 for all comparisons except cataract with ocular symptoms, psychosocial well-being, and role performance scales).

CONCLUSIONS

This vision-targeted HRQOL measure provides a basis for comprehensive assessment of the impact of eye diseases and treatments on daily functioning and well-being in adults.

Norming plans for the NIH Toolbox

OBJECTIVE

The NIH Toolbox for Assessment of Neurological and Behavioral Function (NIH Toolbox) is a comprehensive battery of brief assessment tools. The purpose of this article is to describe plans to establish normative reference values for the NIH Toolbox measures.

METHODS

A large sample will be obtained from the US population for the purpose of calculating normative values. The sample will be stratified by age (ages 3-85 years), sex, and language preference (English or Spanish) and have a total sample size of at least 4,205. The sample will include a minimum of 25-100 individuals in each targeted demographic and language subgroup.

RESULTS

Norming methods will include poststratification adjustment calculated using iterative proportional fitting, also known as raking, so that the weighted sample will have the same distribution on key demographic variables as the US population described in the 2010 Census.

CONCLUSIONS

As with any set of norms, users should be mindful of the reference population and make conclusions consistent with the limitations of normative sampling, since it is not a probability-based sample. However, the NIH Toolbox norming study has been designed to minimize bias and maximize representativeness and precision of estimates. The availability of a "toolbox" of normed measures will be an important foundation for addressing critical research questions in neurologic and behavioral health.

Using the NIH Toolbox in special populations: considerations for assessment of pediatric, geriatric, culturally diverse, non-English-speaking, and disabled individuals

BACKGROUND

In order to develop health outcomes measures that are relevant and applicable to the general population, it is essential to consider the needs and requirements of special subgroups, such as the young, elderly, disabled, and people of different ethnic and cultural backgrounds, within that population.

METHODS

The NIH Toolbox project convened several working groups to address assessment issues for the following subgroups: pediatric, geriatric, cultural, non-English-speaking, and disabled. Each group reviewed all NIH Toolbox instruments in their entirety.

RESULTS

Each working group provided recommendations to the scientific study teams regarding instrument content, presentation, and administration. When feasible and appropriate, instruments and administration procedures have been modified in accordance with these recommendations.

CONCLUSION

Health outcome measurement can benefit from expert input regarding assessment considerations for special subgroups.

ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research

PURPOSE

An essential aspect of patient-centered outcomes research (PCOR) and comparative effectiveness research (CER) is the integration of patient perspectives and experiences with clinical data to evaluate interventions. Thus, PCOR and CER require capturing patient-reported outcome (PRO) data appropriately to inform research, healthcare delivery, and policy. This initiative's goal was to identify minimum standards for the design and selection of a PRO measure for use in PCOR and CER.

METHODS

We performed a literature review to find existing guidelines for the selection of PRO measures. We also conducted an online survey of the International Society for Quality of Life Research (ISOQOL) membership to solicit input on PRO standards. A standard was designated as "recommended" when >50 % respondents endorsed it as "required as a minimum standard."

RESULTS

The literature review identified 387 articles. Survey response rate was 120 of 506 ISOQOL members. The respondents had an average of 15 years experience in PRO research, and 89 % felt competent or very competent providing feedback. Final recommendations for PRO measure standards included: documentation of the conceptual and measurement model; evidence for reliability, validity (content validity, construct validity, responsiveness); interpretability of scores; quality translation, and acceptable patient and investigator burden.

CONCLUSION

The development of these minimum measurement standards is intended to promote the appropriate use of PRO measures to inform PCOR and CER, which in turn can improve the effectiveness and efficiency of healthcare delivery. A next step is to expand these minimum standards to identify best practices for selecting decision-relevant PRO measures.

Neuro-QOL: brief measures of health-related quality of life for clinical research in neurology

OBJECTIVE

To address the need for brief, reliable, valid, and standardized quality of life (QOL) assessment applicable across neurologic conditions.

METHODS

Drawing from larger calibrated item banks, we developed short measures (8-9 items each) of 13 different QOL domains across physical, mental, and social health and evaluated their validity and reliability. Three samples were utilized during short form development: general population (Internet-based, n = 2,113); clinical panel (Internet-based, n = 553); and clinical outpatient (clinic-based, n = 581). All short forms are expressed as T scores with a mean of 50 and SD of 10.

RESULTS

Internal consistency (Cronbach α) of the 13 short forms ranged from 0.85 to 0.97. Correlations between short form and full-length item bank scores ranged from 0.88 to 0.99 (0.82-0.96 after removing common items from banks). Online respondents were asked whether they had any of 19 different chronic health conditions, and whether or not those reported conditions interfered with ability to function normally. All short forms, across physical, mental, and social health, were able to separate people who reported no health condition from those who reported 1-2 or 3 or more. In addition, scores on all 13 domains were worse for people who acknowledged being limited by the health conditions they reported, compared to those who reported conditions but were not limited by them.

CONCLUSION

These 13 brief measures of self-reported QOL are reliable and show preliminary evidence of concurrent validity inasmuch as they differentiate people based upon number of reported health conditions and whether those reported conditions impede normal function.