Successful validation of the CAT-MH Scales in a sample of Latin American migrants in the United States and Spain

Adaptive measurement of cognitive function based on multidimensional item response theory

INTRODUCTION

Up to 20% of older adults in the United States have mild cognitive impairment (MCI), and about one-third of people with MCI are predicted to transition to Alzheimer's disease (AD) within 5 years. Standard cognitive assessments are long and require a trained technician to administer. We developed the first computerized adaptive test (CAT) based on multidimensional item response theory (MIRT) to more precisely, rapidly, and repeatedly assesses cognitive abilities across the adult lifespan. We present results for a prototype CAT (pCAT-COG) for assessment of global cognitive function.

METHODS

We sampled items across five cognitive domains central to neuropsychological testing (episodic memory [EM], semantic memory/language [SM], working memory [WM], executive function/flexible thinking, and processing speed [PS]). The item bank consists of 54 items, with 9 items of varying difficulty drawn from six different cognitive tasks. Each of the 54 items has 3 response trials, yielding an ordinal score (0-3 trials correct). We also include three long-term memory items not designed for adaptive administration, for a total bank of 57 items. Calibration data were collected in-person and online, calibrated using a bifactor MIRT model, and pCAT-COG scores validated against a technician-administered neuropsychological battery.

RESULTS

The bifactor MIRT model improved fit over a unidimensional IRT model (p < 0.0001). The global pCAT-COG scores were inversely correlated with age (r = -0.44, p < 0.0001). Simulated adaptive administration of 11 items maintained a correlation of r = 0.94 with the total item bank scores. Significant differences between mild and no cognitive impairment (NCI) were found (effect size of 1.08 SD units). The pCAT-COG correlated with clinician-based global measure (r = 0.64).

DISCUSSION

MIRT-based CAT is feasible and valid for the assessment of global cognitive impairment, laying the foundation for the development of a full CAT-COG that will draw from a much larger item bank with both global and domain specific measures of cognitive impairment.

Highlights

As Americans age, numbers at risk for developing cognitive impairment are increasing.Aging-related declines in cognition begins decades prior to the onset of obvious cognitive impairment.Traditional assessment is burdensome and requires trained clinicians.We developed an adaptive testing framework using multidimensional item response theory.It is comparable to lengthier in-person assessments that require trained psychometrists.

Undergraduate class standing, perceived social support, and depressive symptoms

OBJECTIVE

Investigate the association between perceived social support and depressive symptoms at different stages of academic progress.

PARTICIPANTS

Undergraduate students (n = 505) enrolled at a large southeastern university.

METHODS

Students completed a cross-sectional survey about their self-reported physical and mental health. Logistic regression was used to assess the relationship between depressive symptoms, perceived social support, and academic class standing.

RESULTS

Academic class standing and perceived social support were both significantly associated with depressive symptoms. Compared to freshman, odds of having depressive symptoms were 2.15 times higher for sophomores and 3.94 times higher for seniors. For every one unit increase in perceived social support, the odds of depressive symptoms decreased by 51%.

CONCLUSIONS

A significant association between depressive symptoms and social support was identified for all undergraduates in this sample. The differences identified between social support and depressive symptoms reveal the need to tailor support provided at different academic stages.

Determinants of remote measurement-based care uptake in a safety net outpatient psychiatry department as part of learning health system transition

Introduction

Behavioral measurement-based care (MBC) can improve patient outcomes and has also been advanced as a critical learning health system (LHS) tool for identifying and mitigating potential disparities in mental health treatment. However, little is known about the uptake of remote behavioral MBC in safety net settings, or possible disparities occurring in remote MBC implementation.

Methods

This study uses electronic health record data to study variation in completion rates at the clinic and patient level of a remote MBC symptom measure tool during the first 6 months of implementation at three adult outpatient psychiatry clinics in a safety net health system. Provider-reported barriers to MBC adoption were also measured using repeated surveys at one of the three sites.

Results

Out of 1219 patients who were sent an MBC measure request, uptake of completing at least one measure varied by clinic: General Adult Clinic, 38% (n = 262 of 696); Substance Use Clinic, 28% (n = 73 of 265); and Transitions Clinic, 17% (n = 44 of 258). Compared with White patients, Black and Portuguese or Brazilian patients had lower uptake. Older patients also had lower uptake. Spanish language of care was associated with much lower uptake at the patient level. Significant patient-level disparities in uptake persisted after adjusting for the clinic, mental health diagnoses, and number of measure requests sent. Providers cited time within visits and bandwidth in their workflow as the greatest consistent barriers to discussing MBC results with patients.

Conclusions

There are significant disparities in MBC uptake at the patient and clinic level. From an LHS data infrastructure perspective, safety net health systems may need to address the need for possible ways to adapt MBC to better fit their populations and clinical needs, or identify targeted implementation strategies to close data gaps for the identified disparity populations.

Validation of the structured interview section of the penn computerized adaptive test for neurocognitive and clinical psychopathology assessment (CAT GOASSESS)

Large-scale studies and burdened clinical settings require precise, efficient measures that assess multiple domains of psychopathology. Computerized adaptive tests (CATs) can reduce administration time without compromising data quality. We examined feasibility and validity of an adaptive psychopathology measure, GOASSESS, in a clinical community-based sample (N = 315; ages 18-35) comprising three groups: healthy controls, psychosis, mood/anxiety disorders. Assessment duration was compared between the Full and CAT GOASSESS. External validity was tested by comparing how the CAT and Full versions related to demographic variables, study group, and socioeconomic status. The relationships between scale scores and criteria were statistically compared within a mixed-model framework to account for dependency between relationships. Convergent validity was assessed by comparing scores of the CAT and the Full GOASSESS using Pearson correlations. The CAT GOASSESS reduced interview duration by more than 90 % across study groups and preserved relationships to external criteria and demographic variables as the Full GOASSESS. All CAT GOASSESS scales could replace those of the Full instrument. Overall, the CAT GOASSESS showed acceptable psychometric properties and demonstrated feasibility by markedly reducing assessment time compared to the Full GOASSESS. The adaptive version could be used in large-scale studies or clinical settings for intake screening.

Referrals to Community and State Agencies to Address Social Determinants of Health for Improving Mental Health, Functioning, and Quality of Care Outcomes for Diverse Adults

Objectives. To examine whether referral for social determinants of health (SDH) needs decreases psychological distress and posttraumatic stress disorder (PTSD) symptoms and improves level of functioning and quality of care among diverse adults. Methods. Data are from control participants (n = 503 adults) in a randomized controlled trial testing a mental health intervention in North Carolina and Massachusetts. We fitted multilevel mixed-effects models to repeated assessments (baseline, 3, 6, and 12 months) collected between September 2019 and January 2023. Results. After referral to services for trouble paying utility bills, participants reported lower PTSD symptoms. Participants reported better quality of care when receiving referrals to mental health care. After adjusting for income and employment status, we found that participants who were referred more often also had lower PTSD symptoms and better levels of functioning. Conclusions. Referrals for certain SDH needs might decrease PTSD symptoms and improve self-reported quality of care and functioning. However, referrals alone, without ensuring receipt of services, might be insufficient to affect other mental health outcomes. Research is needed on training and providing care managers time for offering interpersonal support, securing services, and understanding agencies' contexts for addressing high SDH needs. (Am J Public Health. 2024;114(S3):S278-S288. https://doi.org/10.2105/AJPH.2023.307442).

Participants' Engagement With and Results From a Web-Based Integrative Population Mental Wellness Program (CHAMindWell) During the COVID-19 Pandemic: Program Evaluation Study

BACKGROUND

The COVID-19 pandemic involved a prolonged period of collective trauma and stress during which substantial increases in mental health concerns, like depression and anxiety, were observed across the population. In this context, CHAMindWell was developed as a web-based intervention to improve resilience and reduce symptom severity among a public health care system's patient population.

OBJECTIVE

This program evaluation was conducted to explore participants' engagement with and outcomes from CHAMindWell by retrospectively examining demographic information and mental health symptom severity scores throughout program participation.

METHODS

We examined participants' symptom severity scores from repeated, web-based symptom screenings through Computerized Adaptive Testing for Mental Health (CAT-MH) surveys, and categorized participants into symptom severity-based tiers (tier 1=asymptomatic to mild; tier 2=moderate; and tier 3=severe). Participants were provided tier-based mindfulness resources, treatment recommendations, and referrals. Logistic regressions were conducted to evaluate associations between demographic variables and survey completion. The McNemar exact test and paired sample t tests were performed to evaluate changes in the numbers of participants in tier 1 versus tier 2 or 3 and changes in depression, anxiety, and posttraumatic stress disorder severity scores between baseline and follow-up.

RESULTS

The program enrolled 903 participants (664/903, 73.5% female; 556/903, 61.6% White; 113/903, 12.5% Black; 84/903, 9.3% Asian; 7/903, 0.8% Native; 36/903, 4% other; and 227/903, 25.1% Hispanic) between December 16, 2020, and March 17, 2022. Of those, 623 (69%) completed a baseline CAT-MH survey, and 196 completed at least one follow-up survey 3 to 6 months after baseline. White racial identity was associated with completing baseline CAT-MH (odds ratio [OR] 1.80, 95% CI 1.14-2.84; P=.01). Participants' odds of having symptom severity below the clinical threshold (ie, tier 1) were significantly greater at follow-up (OR 2.60, 95% CI 1.40-5.08; P=.001), and significant reductions were observed across symptom domains over time.

CONCLUSIONS

CHAMindWell is associated with reduced severity of mental health symptoms. Future work should aim to address program engagement inequities and attrition and compare the impacts of CHAMindWell to a control condition to better characterize its effects.

eHealth Familias Unidas Mental Health: Protocol for an effectiveness-implementation hybrid Type 1 trial to scale a mental health preventive intervention for Hispanic youth in primary care settings

This article focuses on the rationale, design and methods of an effectiveness-implementation hybrid type I randomized trial of eHealth Familias Unidas Mental Health, a family-based, online delivered intervention for Hispanic families to prevent/reduce depressive and anxious symptoms, suicide ideation/behaviors, and drug use in Hispanic youth. Utilizing a rollout design with 18 pediatric primary care clinics and 468 families, this study addresses intervention effectiveness, implementation research questions, and intervention sustainment, to begin bridging the gap between research and practice in eliminating mental health and drug use disparities among Hispanic youth. Further, we will examine whether intervention effects are partially mediated by improved family communication and reduced externalizing behaviors, including drug use, and moderated by parental depression. Finally, we will explore whether the intervention's impact on mental health and drug use, as well as sustainment of the intervention in clinics, varies by quality of implementation at clinic and clinician levels. Trail registration: ClinicalTrials.gov Identifier: NCT05426057, First posted June 21, 2022.

Role of social determinants in anxiety and depression symptoms during COVID-19: A longitudinal study of adults in North Carolina and Massachusetts

Trajectory studies of the COVID-19 pandemic have described patterns of symptoms over time. Yet, few have examined whether social determinants of health predict the progression of depression and anxiety symptoms during COVID-19 or identified which social determinants worsen symptom trajectories. Using a racially, ethnically, and linguistically diverse sample of adults participating in a randomized clinical trial with pre-existing moderate to severe depression and/or anxiety symptoms, we compare symptom patterns before and during COVID-19; characterize symptom trajectories over a 20-week follow-up period; and evaluate whether social determinants are associated with within- and between- person differences in symptom trajectories. Data were collected before and during COVID-19 in Massachusetts and North Carolina. On average, depression and anxiety symptoms did not seem to worsen during the pandemic compared to pre-pandemic. During COVID-19, anxiety scores at follow-up were higher for participants with baseline food insecurity (vs no food insecurity). Depression scores at follow-up were higher for participants with food insecurity and for those with utilities insecurity (vs no insecurity). Participants with child or family care responsibilities at baseline had depression symptoms decreasing at a slower rate than those without these responsibilities. We discuss the important implications of these findings.

Development and Validation of Computerized Adaptive Assessment Tools for the Measurement of Posttraumatic Stress Disorder Among US Military Veterans

IMPORTANCE

Veterans from recent and past conflicts have high rates of posttraumatic stress disorder (PTSD). Adaptive testing strategies can increase accuracy of diagnostic screening and symptom severity measurement while decreasing patient and clinician burden.

OBJECTIVE

To develop and validate a computerized adaptive diagnostic (CAD) screener and computerized adaptive test (CAT) for PTSD symptom severity.

DESIGN, SETTING, AND PARTICIPANTS

A diagnostic study of measure development and validation was conducted at a Veterans Health Administration facility. A total of 713 US military veterans were included. The study was conducted from April 25, 2017, to November 10, 2019.

MAIN OUTCOMES AND MEASURES

The participants completed a PTSD-symptom questionnaire from the item bank and provided responses on the PTSD Checklist for Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) (PCL-5). A subsample of 304 participants were interviewed using the Clinician-Administered Scale for PTSD for DSM-5.

RESULTS

Of the 713 participants, 585 were men; mean (SD) age was 52.8 (15.0) years. The CAD-PTSD reproduced the Clinician-Administered Scale for PTSD for DSM-5 PTSD diagnosis with high sensitivity and specificity as evidenced by an area under the curve of 0.91 (95% CI, 0.87-0.95). The CAT-PTSD demonstrated convergent validity with the PCL-5 (r = 0.88) and also tracked PTSD diagnosis (area under the curve = 0.85; 95% CI, 0.79-0.89). The CAT-PTSD reproduced the final 203-item bank score with a correlation of r = 0.95 with a mean of only 10 adaptively administered items, a 95% reduction in patient burden.

CONCLUSIONS AND RELEVANCE

Using a maximum of only 6 items, the CAD-PTSD developed in this study was shown to have excellent diagnostic screening accuracy. Similarly, using a mean of 10 items, the CAT-PTSD provided valid severity ratings with excellent convergent validity with an extant scale containing twice the number of items. The 10-item CAT-PTSD also outperformed the 20-item PCL-5 in terms of diagnostic accuracy. The results suggest that scalable, valid, and rapid PTSD diagnostic screening and severity measurement are possible.

Development and Validation of a Computerized Adaptive Assessment Tool for Discrimination and Measurement of Psychotic Symptoms

OBJECTIVE

Time constraints limit the use of measurement-based approaches in research and routine clinical management of psychosis. Computerized adaptive testing (CAT) can reduce administration time, thus increasing measurement efficiency. This study aimed to develop and test the capacity of the CAT-Psychosis battery, both self-administered and rater-administered, to measure the severity of psychotic symptoms and discriminate psychosis from healthy controls.

METHODS

An item bank was developed and calibrated. Two raters administered CAT-Psychosis for inter-rater reliability (IRR). Subjects rated themselves and were retested within 7 days for test-retest reliability. The Brief Psychiatric Rating Scale (BPRS) was administered for convergent validity and chart diagnosis, and the Structured Clinical Interview (SCID) was used to test psychosis discriminant validity.

RESULTS

Development and calibration study included 649 psychotic patients. Simulations revealed a correlation of r = .92 with the total 73-item bank score, using an average of 12 items. Validation study included 160 additional patients and 40 healthy controls. CAT-Psychosis showed convergent validity (clinician: r = 0.690; 95% confidence interval [95% CI]: 0.610-0.757; self-report: r = .690; 95% CI: 0.609-0.756), IRR (intraclass correlation coefficient [ICC] = 0.733; 95% CI: 0.611-0.828), and test-retest reliability (clinician ICC = 0.862; 95% CI: 0.767-0.922; self-report ICC = 0.815; 95%CI: 0.741-0.871). CAT-Psychosis could discriminate psychosis from healthy controls (clinician: area under the receiver operating characteristic curve [AUC] = 0.965, 95% CI: 0.945-0.984; self-report AUC = 0.850, 95% CI: 0.807-0.894). The median length of the clinician-administered assessment was 5 minutes (interquartile range [IQR]: 3:23-8:29 min) and 1 minute, 20 seconds (IQR: 0:57-2:09 min) for the self-report.

CONCLUSION

CAT-Psychosis can quickly and reliably assess the severity of psychosis and discriminate psychotic patients from healthy controls, creating an opportunity for frequent remote assessment and patient/population-level follow-up.

Computerized Adaptive Tests for Rapid and Accurate Assessment of Psychopathology Dimensions in Youth

OBJECTIVE

At least half of youths with mental disorders are unrecognized and untreated. Rapid, accurate assessment of child mental disorders could facilitate identification and referral and potentially reduce the occurrence of functional disability that stems from early-onset mental disorders.

METHOD

Computerized adaptive tests (CATs) based on multidimensional item response theory were developed for depression, anxiety, mania/hypomania, attention-deficit/hyperactivity disorder, conduct disorder, oppositional defiant disorder, and suicidality, based on parent and child ratings of 1,060 items each. In phase 1, CATs were developed from 801 participants. In phase 2, predictive, discriminant, and convergent validity were tested against semi-structured research interviews for diagnoses and suicidality in 497 patients and 104 healthy controls. Overall strength of association was determined by area under the receiver operating characteristic curve (AUC).

RESULTS

The child and parent independently completed the Kiddie-Computerized Adaptive Tests (K-CATs) in a median time of 7.56 and 5.03 minutes, respectively, with an average of 7 items per domain. The K-CATs accurately captured the presence of diagnoses (AUCs from 0.83 for generalized anxiety disorder to 0.92 for major depressive disorder) and suicidal ideation (AUC = 0.996). Strong correlations with extant measures were found (r ≥ 0.60). Test-retest reliability averaged r = 0.80.

CONCLUSION

These K-CATs provide a new approach to child psychopathology screening and measurement. Testing can be completed by child and parent in less than 8 minutes and yields results that are highly convergent with much more time-consuming structured clinical interviews and dimensional severity assessment and measurement. Testing of the implementation of the K-CAT is now indicated.

Improving the Evaluation of Adult Mental Disorders in the Criminal Justice System With Computerized Adaptive Testing

OBJECTIVE

The authors sought to develop and validate a suite of dimensional measures of psychiatric syndromes for use in a criminal justice population.

METHODS

The previously validated Computerized Adaptive Test-Mental Health (CAT-MH) was administered to a sample of 475 defendants in the Cook County Bond Court. Item-level data were used to determine which test items exhibited differential item functioning in this population compared with the population used for the original calibration.

RESULTS

After removal of nine items that exhibited differential item functioning from the CAT-MH, correlations between scores based on the original calibration from a nonjustice-involved population and the newly computed scores based on a sample of bond court defendants showed a correlation coefficient of r=0.96 to r=0.99.

CONCLUSIONS

With a slight modification of the original CAT-MH, the tool was successfully used to measure severity of depression, anxiety, mania and/or hypomania, suicidality, and substance use disorder in an English- and Spanish-speaking criminal justice population.

Without Wasting a Word: Extreme Improvements in Efficiency and Accuracy Using Computerized Adaptive Testing for Mental Health Disorders (CAT-MH)

PURPOSE OF REVIEW

We review recent literature on the adaptive assessment of complex mental health disorders and provide a detailed comparison of classical test theory and adaptive testing based on multidimensional item response theory.

RECENT FINDINGS

Adaptive tests for a wide variety of mental health traits (e.g., depression, anxiety, mania, substance misuse, suicidality) are now available in a cloud-based environment. These tests have been validated in a variety of settings against lengthy structured clinical interviews with excellent results and even higher reliability than fixed-length tests. Applications include screening and assessments in emergency departments, psychiatric and primary care clinics, student health clinics, perinatal medicine clinics, child welfare settings, and the judicial system. The future of mental health measurement will be based on automated screening and assessments. Adaptive tests will provide increased precision of measurement and decreased burden of measurement. Integration into the electronic health record is important and now easily accomplished.