Developing and Validating a Computerized Adaptive Test to Measure Broad and Specific Factors of Internalizing in a Community Sample

Developing an accurate and efficient tool for the internalizing spectrum: A simulation study of the adaptive algorithm to the Inventory of Depression and Anxiety Symptoms II (IDAS-II)

OBJECTIVES

This research simulates an adaptive version of the IDAS-II (IDAS-CAT).

METHODS

2021 participants from both community (n = 1692) and patients (n = 329) samples completed the IDAS-II. Item response theory metric properties of the IDAS-II full test and the 20-items of the general depression (GD) scale were obtained. The efficiency and accuracy of different computerized adaptive algorithms were simulated. Different subsamples completed additional external measures in order to gather evidence of validity of the scores estimated with the simulated adaptive algorithms selected.

RESULTS

Both unidimensional computerized adaptive testing algorithm selected for the GD scale and the bifactor model chosen for the full test, allow 70% reduction in the length of administration, maintaining a measurement error below 0.30 on the general and 0.50 on the specific factors. Results show high correlations of the scores estimated with the adaptive algorithms and the estimates based on the full test, as well as correlations with external criteria almost equal to those generated with the full test.

CONCLUSIONS

IDAS-CAT could be a reliable and fast tool for measuring internalizing spectrum.

Validation of the structured interview section of the penn computerized adaptive test for neurocognitive and clinical psychopathology assessment (CAT GOASSESS)

Large-scale studies and burdened clinical settings require precise, efficient measures that assess multiple domains of psychopathology. Computerized adaptive tests (CATs) can reduce administration time without compromising data quality. We examined feasibility and validity of an adaptive psychopathology measure, GOASSESS, in a clinical community-based sample (N = 315; ages 18-35) comprising three groups: healthy controls, psychosis, mood/anxiety disorders. Assessment duration was compared between the Full and CAT GOASSESS. External validity was tested by comparing how the CAT and Full versions related to demographic variables, study group, and socioeconomic status. The relationships between scale scores and criteria were statistically compared within a mixed-model framework to account for dependency between relationships. Convergent validity was assessed by comparing scores of the CAT and the Full GOASSESS using Pearson correlations. The CAT GOASSESS reduced interview duration by more than 90 % across study groups and preserved relationships to external criteria and demographic variables as the Full GOASSESS. All CAT GOASSESS scales could replace those of the Full instrument. Overall, the CAT GOASSESS showed acceptable psychometric properties and demonstrated feasibility by markedly reducing assessment time compared to the Full GOASSESS. The adaptive version could be used in large-scale studies or clinical settings for intake screening.

Measuring Process Factors of Fluid Reasoning Using Multidimensional Computerized Adaptive Testing

Although many fluid reasoning (Gf) tests have been developed, there is a lack of figural tests measuring its lower-order process factors simultaneously. The present article introduces the development of the Multidimensional Induction-Deduction Computerized Adaptive Test (MID-CAT) to measure two process factors of Gf. The MID-CAT is designed to provide an instrument that is flexible, efficient, and entirely free for non-commercial use. We created 530 items and administered them to a sample of N = 2,247. Items were fitted and calibrated using the Rasch model. The results indicate that the final item pool has a wide range of difficulties that could precisely measure a wide range of test-takers' abilities. A simulation study also indicates that MID-CAT provides greater measurement efficiency than separate-unidimensional CAT or fixed-item test. In the discussion, we provide perspectives on how the MID-CAT can be used for future research.

Applying Item Response Theory to the Student Adaptation to College Questionnaire: Examining Psychometric Characteristics and Developing Computerized Adaptive Testing Version

Incoming students have many difficulties adjusting to college, and selecting appropriate measures to effectively screen them is indispensable, especially in China, where there is insufficient research in this area. To enrich domestic research, this study seeks to examine psychometric characteristics and develop a computerized adaptive version of the Student Adaptation to College Questionnaire (SACQ-CAT) based on a sample of Chinese students. Under the framework of item response theory, the item bank of student adaptation to college was formulated after uni-dimensionality testing, model comparison, item fit testing, and local independence testing. Subsequently, a CAT simulation, including three termination rules, was performed using real data to evaluate and verify the SACQ-CAT. The results showed reliability values exceeding 0.90 when participants' latent traits were between -4 and 3, covering majority of the subjects. The SACQ-CAT administered an average of fewer than 10 items to participants compared to 67 items on the original scale. The correlation coefficient between latency estimated by the SACQ-CAT and the SACQ is greater than .85, whereas the correlation coefficient with the Symptom Checklist 90 (SCL-90) scores ranges from -.33 to -.55 (p < .001). The SACQ-CAT largely reduced the number of items administered to the participants and avoided losing measurement precision.

Applying multidimensional computerized adaptive testing to the MSQOL-54: a simulation study

BACKGROUND

The Multiple Sclerosis Quality of Life-54 (MSQOL-54) is one of the most commonly-used MS-specific health-related quality of life (HRQOL) measures. It is a multidimensional, MS-specific HRQOL inventory, which includes the generic SF-36 core items, supplemented with 18 MS-targeted items. Availability of an adaptive short version providing immediate item scoring may improve instrument usability and validity. However, multidimensional computerized adaptive testing (MCAT) has not been previously applied to MSQOL-54 items. We thus aimed to apply MCAT to the MSQOL-54 and assess its performance.

METHODS

Responses from a large international sample of 3669 MS patients were assessed. We calibrated 52 (of the 54) items using bifactor graded response model (10 group factors and one general HRQOL factor). Then, eight simulations were run with different termination criteria: standard errors (SE) for the general factor and group factors set to different values, and change in factor estimates from one item to the next set at < 0.01 for both the general and the group factors. Performance of the MCAT was assessed by the number of administered items, root mean square difference (RMSD), and correlation.

RESULTS

Eight items were removed due to local dependency. The simulation with SE set to 0.32 (general factor), and no SE thresholds (group factors) provided satisfactory performance: the median number of administered items was 24, RMSD was 0.32, and correlation was 0.94.

CONCLUSIONS

Compared to the full-length MSQOL-54, the simulated MCAT required fewer items without losing precision for the general HRQOL factor. Further work is needed to add/integrate/revise MSQOL-54 items in order to make the calibration and MCAT performance efficient also on group factors, so that the MCAT version may be used in clinical practice and research.

Bayesian modeling of item heterogeneity in dichotomous recognition memory data and prospects for computerized adaptive testing

Most current models of recognition memory fail to separately model item and person heterogeneity which makes it difficult to assess ability at the latent construct level and prevents the administration of adaptive tests. Here we propose to employ a General Condorcet Model for Recognition (GCMR) in order to estimate ability, response bias and item difficulty in dichotomous recognition memory tasks. Using a Bayesian modeling framework and MCMC inference, we perform 3 separate validation studies comparing GCMR to the Rasch model from IRT and the 2-High-Threshold (2HT) recognition model. First, two simulations demonstrate that recovery of GCMR ability estimates with varying sparsity and test difficulty is more robust and that estimates improve from the two other models under common test scenarios. Then, using a real dataset, face validity is confirmed by replicating previous findings of general and domain-specific age effects (Güsten et al. in Cortex 137:138-148, https://doi.org/10.1016/j.cortex.2020.12.017 , 2021). Using cross-validation we show better out-of-sample prediction for the GCMR as compared to Rasch and 2HT model. In addition, we present a hierarchical extension of the model that is able to estimate age- and domain-specific effects directly, without recurring to a two-stage procedure. Finally, an adaptive test using the GCMR is simulated, showing that the test length necessary to obtain reliable ability estimates can be significantly reduced compared to a non-adaptive procedure. The GCMR allows to model trial-by-trial performance and to increase the efficiency and reliability of recognition memory assessments.

A detailed hierarchical model of psychopathology: From individual symptoms up to the general factor of psychopathology

Much of our knowledge about the relationships among domains of psychopathology is built on the diagnostic categories described in the Diagnostic and Statistical Manual of Mental Disorders (DSM), with relatively little research examining the symptom-level structure of psychopathology. The aim of this study was to delineate a detailed hierarchical model of psychopathology-from individual symptoms up to a general factor of psychopathology-allowing both higher- and lower-order dimensions to depart from the structure of the DSM. We explored the hierarchical structure of hundreds of symptoms spanning 18 DSM disorders, in two large samples-one from the general population in Australia (n = 3175), and the other a treatment-seeking clinical sample from the USA (n = 1775). There was marked convergence between the two samples, offering new perspectives on higher-order dimensions of psychopathology. We also found several noteworthy departures from the structure of the DSM in the symptom-level data.

Development of the RMT20, a composite screener to identify common mental disorders

BACKGROUND

There are few very brief measures that accurately identify multiple common mental disorders.

AIMS

The aim of this study was to develop and assess the psychometric properties of a new composite measure to screen for five common mental disorders.

METHOD

Two cross-sectional psychometric surveys were used to develop (n = 3175) and validate (n = 3620) the new measure, the Rapid Measurement Toolkit-20 (RMT20) against diagnostic criteria. The RMT20 was tested against a DSM-5 clinical checklist for major depression, generalised anxiety disorder, panic disorder, social anxiety disorder and post-traumatic stress disorder, with comparison with two measures of general psychological distress: the Kessler-10 and Distress Questionnaire-5.

RESULTS

The area under the curve for the RMT20 was significantly greater than for the distress measures, ranging from 0.86 to 0.92 across the five disorders. Sensitivity and specificity at prescribed cut-points were excellent, with sensitivity ranging from 0.85 to 0.93 and specificity ranging from 0.73 to 0.83 across the five disorders.

CONCLUSIONS

The RMT20 outperformed two established scales assessing general psychological distress, is free to use and has low respondent burden. The measure is well-suited to clinical screening, internet-based screening and large-scale epidemiological surveys.

Comparing Scores From Full Length, Short Form, and Adaptive Tests of the Social Interaction Anxiety and Social Phobia Scales

The current study developed and examined the performance of a computerized adaptive version of the Social Interaction Anxiety and Social Phobia Scales (SIAS/SPS) and compared results with a previously developed static short form (SIAS-6/SPS-6) in terms of measurement precision, concordance with the full forms, and sensitivity to treatment. Among an online sample of Australian adults, there were relatively minor differences in the performance of the adaptive tests and static short forms when compared with the full scales. Moreover, both adaptive and static short forms generated similar effect sizes across treatment in a clinical sample. This provides further evidence for the use of static or adaptive short forms of the SIAS/SPS rather than the lengthier 20-item versions. However, at the individual level, the adaptive tests were able to maintain an acceptable level of precision, using few items as possible, across the severity continua in contrast to the static short forms.

Measurement Efficiency for Fixed-Precision Multidimensional Computerized Adaptive Tests: Comparing Health Measurement and Educational Testing Using Example Banks

It is currently not entirely clear to what degree the research on multidimensional computerized adaptive testing (CAT) conducted in the field of educational testing can be generalized to fields such as health assessment, where CAT design factors differ considerably from those typically used in educational testing. In this study, the impact of a number of important design factors on CAT performance is systematically evaluated, using realistic example item banks for two main scenarios: health assessment (polytomous items, small to medium item bank sizes, high discrimination parameters) and educational testing (dichotomous items, large item banks, small- to medium-sized discrimination parameters). Measurement efficiency is evaluated for both between-item multidimensional CATs and separate unidimensional CATs for each latent dimension. In this study, we focus on fixed-precision (variable-length) CATs because it is both feasible and desirable in health settings, but so far most research regarding CAT has focused on fixed-length testing. This study shows that the benefits associated with fixed-precision multidimensional CAT hold under a wide variety of circumstances.

Validation of the Computerized Adaptive Test for Mental Health in Primary Care

PURPOSE

The US Preventive Services Task Force recommends screening for depression in the general adult population. Although screening questionnaires for depression and anxiety exist in primary care settings, electronic health tools such as computerized adaptive tests based on item response theory can advance screening practices. This study evaluated the validity of the Computerized Adaptive Test for Mental Health (CAT-MH) for screening for major depressive disorder (MDD) and assessing MDD and anxiety severity among adult primary care patients.

METHODS

We approached 402 English-speaking adults for participation from a primary care clinic, of whom 271 adults (71% female, 65% black) participated. Participants completed modules from the CAT-MH (Computerized Adaptive Diagnostic Test for MDD, CAT-Depression Inventory, CAT-Anxiety Inventory); brief paper questionnaires (9-item Patient Health Questionnaire [PHQ-9], 2-item Patient Health Questionnaire [PHQ-2], Generalized Anxiety Disorder 7-item Scale [GAD-7]); and a reference-standard interview, the Structured Clinical Interview for DSM-5 (Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition) Diagnoses.

RESULTS

On the basis of the interview, 31 participants met criteria for MDD and 29 met criteria for GAD. The diagnostic accuracy of the Computerized Adaptive Diagnostic Test for MDD (area under curve [AUC] = 0.85) was similar to that of the PHQ-9 (AUC = 0.84) and higher than that of the PHQ-2 (AUC = 0.76) for MDD screening. Using the interview as the reference standard, the accuracy of the CAT-Anxiety Inventory (AUC = 0.93) was similar to that of the GAD-7 (AUC = 0.97) for assessing anxiety severity. The patient-preferred screening method was assessment via tablet/computer with audio.

CONCLUSIONS

Computerized adaptive testing could be a valid and efficient patient-centered screening strategy for depression and anxiety screening in primary care settings.

Self-Reported Sleep Quality Modulates Amygdala Resting-State Functional Connectivity in Anxiety and Depression

Sufficient sleep plays an important role in neurocognitive function, yet, problematic sleep is ubiquitous in the general population. It is also frequently predictive of, and concurrent with, internalizing psychopathologies (IPs) such as anxiety and depression suggesting sleep quality is dimensional and transdiagnostic. Along with problematic sleep, IPs are characterized by negative affectivity, therefore, prominent neurobiological models of internalizing conditions involve the amygdala, a region central to emotion. In resting-state studies (independent of sleep considerations), abnormalities in amygdala-frontal functional connectivity are commonly observed suggesting emotion dysregulation may contribute to clinically-relevant phenotypes. In a separate line of research, studies of sleep deprivation, and insomnia disorder suggest sleep loss may alter amygdala-frontal connectivity. Taken together, findings point to shared neurobiology between sleep and emotion systems, however, the impact of sleep quality on the amygdala circuit in anxiety or depression is unclear. Therefore, we evaluated variance in naturalistic sleep quality on amygdala-based circuity in individuals with and without psychiatric illness. Resting-state fMRI data was collected in 87 un-medicated, treatment-seeking adults diagnosed with a primary anxiety disorder (n = 68) or primary depressive disorder (n = 19) in addition to healthy individuals (n = 40). Regression analysis was conducted with bilateral anatomical amygdala as seed regions and self-reported sleep quality was indexed with a validated self-report measure, the Pittsburgh Sleep Quality Index (PSQI). Post-hoc analysis was performed to evaluate whether diagnostic status (primary anxiety, primary depression, healthy) significantly explained functional connectivity results. Whole-brain regression analysis, controlling for anxiety and depression symptoms, revealed worse sleep quality (i.e., higher PSQI total scores) predicted increased left amygdala-subgenual anterior cingulate functional connectivity and reduced connectivity with posterior cerebellar lobe and superior temporal gyrus. For right amygdala, increased coupling with postcentral gyrus corresponded with worse sleep. Post-hoc analysis did not detect a significant relationship between diagnostic status and whole-brain findings. Results expand on previous studies and indicate variance in sleep quality tracks brain pathways involved in cognitive-emotion functions implicated in the neurobiology of IPs that may extend to individuals at risk for clinical anxiety or depression. Altogether, the clinical relevance of identifying phenotypes to improve our understanding of psychopathology may be improved by incorporating sleep quality.