Extensions of Multiple-Group Item Response Theory Alignment: Application to Psychiatric Phenotypes in an International Genomics Consortium

From the Individual to the Group: Using Idiographic Analyses and Two-Stage Random Effects Meta-Analysis to Obtain Population Level Inferences for within-Person Processes

In psychology, the use of portable technology and wearable devices to ease participant burden in data collection is on the rise. This creates increased interest in collecting real-time or near real-time data from individuals within their natural environments. As a result, vast amounts of observational time series data are generated. Often, motivation for collecting this data hinges on understanding within-person processes that underlie psychological phenomena. Motivated by the body of Dr. Peter Molenaar's life work calling for analytical approaches that consider potential heterogeneity and non-ergodicity, the focus of this paper is on using idiographic analyses to generate population inferences for within-person processes. Meta-analysis techniques using one-stage and two-stage random effects meta-analysis as implemented in single-case experimental designs are presented. The case for preferring a two-stage approach for meta-analysis of single-subject observational time series data is made and demonstrated using an empirical example. This provides a novel implementation of the methodology as prior implementations focus on applications to short time series with experimental designs. Inspired by Dr. Molenaar's work, we describe how an approach, two-stage random effects meta-analysis (2SRE-MA), aligns with recent calls to consider idiographic approaches when making population-level inferences regarding within-person processes.

Phenotype harmonization and analysis for The Populations Underrepresented in Mental illness Association Studies (the PUMAS Project)

Background

The Populations Underrepresented in Mental illness Association Studies (PUMAS) project is attempting to remediate the historical underrepresentation of African and Latin American populations in psychiatric genetics through large-scale genetic association studies of individuals diagnosed with a serious mental illness [SMI, including schizophrenia (SCZ), schizoaffective disorder (SZA) bipolar disorder (BP), and severe major depressive disorder (MDD)] and matched controls. Given growing evidence indicating substantial symptomatic and genetic overlap between these diagnoses, we sought to enable transdiagnostic genetic analyses of PUMAS data by conducting phenotype alignment and harmonization for 89,320 participants (48,165 cases and 41,155 controls) from four cohorts, each of which used different ascertainment and assessment methods: PAISA n=9,105; PUMAS-LATAM n=14,638; NGAP n=42,953 and GPC n=22,624. As we describe here, these efforts have yielded harmonized datasets enabling us to analyze PUMAS genetic variation data at three levels: SMI overall, diagnoses, and individual symptoms.

Methods

In aligning item-level phenotypes obtained from 14 different clinical instruments, we incorporated content, branching nature, and time frame for each phenotype; standardized diagnoses; and selected 19 core SMI item-level phenotypes for analyses. The harmonization was evaluated in PUMAS cases using multiple correspondence analysis (MCA), co-occurrence analyses, and item-level endorsement.

Outcomes

We mapped >6,895 item-level phenotypes in the aggregated PUMAS data, in which SCZ (44.97%) and severe BP (BP-I, 31.53%) were the most common diagnoses. Twelve of the 19 core item-level phenotypes occurred at frequencies of > 10% across all diagnoses, indicating their potential utility for transdiagnostic genetic analyses. MCA of the 14 phenotypes that were present for all cohorts revealed consistency across cohorts, and placed MDD and SCZ into separate clusters, while other diagnoses showed no significant phenotypic clustering.

Interpretation

Our alignment strategy effectively aggregated extensive phenotypic data obtained using diverse assessment tools. The MCA yielded dimensional scores which we will use for genetic analyses along with the item level phenotypes. After successful harmonization, residual phenotypic heterogeneity between cohorts reflects differences in branching structure of diagnostic instruments, recruitment strategies, and symptom interpretation (due to cultural variation).

Approaches to protocol standardization and data harmonization in the ECHO-wide cohort study

The United States (U.S.) National Institutes of Health-funded Environmental influences on Child Health Outcomes (ECHO)-wide Cohort was established to conduct high impact, transdisciplinary science to improve child health and development. The cohort is a collaborative research design in which both extant and new data are contributed by over 57,000 children across 69 cohorts. In this review article, we focus on two key challenging issues in the ECHO-wide Cohort: data collection standardization and data harmonization. Data standardization using a Common Data Model and derived analytical variables based on a team science approach should facilitate timely analyses and reduce errors due to data misuse. However, given the complexity of collaborative research designs, such as the ECHO-wide Cohort, dedicated time is needed for harmonization and derivation of analytic variables. These activities need to be done methodically and with transparency to enhance research reproducibility. IMPACT: Many collaborative research studies require data harmonization either prior to analyses or in the analyses of compiled data. The Environmental influences on Child Health Outcomes (ECHO) Cohort pools extant data with new data collection from over 57,000 children in 69 cohorts to conduct high-impact, transdisciplinary science to improve child health and development, and to provide a national database and biorepository for use by the scientific community at-large. We describe the tools, systems, and approaches we employed to facilitate harmonized data for impactful analyses of child health outcomes.

Measurement bias in caregiver-report of early childhood behavior problems across demographic factors in an ECHO-wide diverse sample

Background

Research and clinical practice rely heavily on caregiver-report measures, such as the Child Behavior Checklist 1.5-5 (CBCL/1.5-5), to gather information about early childhood behavior problems and to screen for child psychopathology. While studies have shown that demographic variables influence caregiver ratings of behavior problems, the extent to which the CBCL/1.5-5 functions equivalently at the item level across diverse samples is unknown.

Methods

Item-level data of CBCL/1.5-5 from a large sample of young children (N = 9087) were drawn from 26 cohorts in the Environmental influences on Child Health Outcomes program. Factor analyses and the alignment method were applied to examine measurement invariance (MI) and differential item functioning (DIF) across child (age, sex, bilingual status, and neurodevelopmental disorders), and caregiver (sex, education level, household income level, depression, and language version administered) characteristics. Child race was examined in sensitivity analyses.

Results

Items with the most impactful DIF across child and caregiver groupings were identified for Internalizing, Externalizing, and Total Problems. The robust item sets, excluding the high DIF items, showed good reliability and high correlation with the original Internalizing and Total Problems scales, with lower reliability for Externalizing. Language version of CBCL administration, education level and sex of the caregiver respondent showed the most significant impact on MI, followed by child age. Sensitivity analyses revealed that child race has a unique impact on DIF over and above socioeconomic status.

Conclusions

The CBCL/1.5-5, a caregiver-report measure of early childhood behavior problems, showed bias across demographic groups. Robust item sets with less DIF can measure Internalizing and Total Problems equally as well as the full item sets, with slightly lower reliability for Externalizing, and can be crosswalked to the metric of the full item set, enabling calculation of normed T scores based on more robust item sets.

An Evaluation of Item Harmonization Strategies Between Assessment Tools of Psychopathology in Children and Adolescents

Data aggregation in mental health is complicated by using different questionnaires, and little is known about the impact of item harmonization strategies on measurement precision. Therefore, we aimed to assess the impact of various item harmonization strategies for a target and proxy questionnaire using correlated and bifactor models. Data were obtained from the Brazilian High-Risk Study for Mental Conditions (BHRCS) and the Healthy Brain Network (HBN; N = 6,140, ages 5-22 years, 39.6% females). We tested six item-wise harmonization strategies and compared them based on several indices. The one-by-one (1:1) expert-based semantic item harmonization presented the best strategy as it was the only that resulted in scalar-invariant models for both samples and factor models. The between-questionnaires factor correlation, reliability, and factor score difference in using a proxy instead of a target measure improved little when all other harmonization strategies were compared with a completely at-random strategy. However, for bifactor models, between-questionnaire specific factor correlation increased from 0.05-0.19 (random item harmonization) to 0.43-0.60 (expert-based 1:1 semantic harmonization) in BHRCS and HBN samples, respectively. Therefore, item harmonization strategies are relevant for specific factors from bifactor models and had little impact on p-factors and first-order correlated factors when the child behavior checklist (CBCL) and strengths and difficulties questionnaire (SDQ) were harmonized.

Harmonizing bifactor models of psychopathology between distinct assessment instruments: Reliability, measurement invariance, and authenticity

OBJECTIVES

Model configuration is important for mental health data harmonization. We provide a method to investigate the performance of different bifactor model configurations to harmonize different instruments.

METHODS

We used data from six samples from the Reproducible Brain Charts initiative (N = 8,606, ages 5-22 years, 41.0% females). We harmonized items from two psychopathology instruments, Child Behavior Checklist (CBCL) and GOASSESS, based on semantic content. We estimated bifactor models using confirmatory factor analysis, and calculated their model fit, factor reliability, between-instrument invariance, and authenticity (i.e., the correlation and factor score difference between the harmonized and original models).

RESULTS

Five out of 12 model configurations presented acceptable fit and were instrument-invariant. Correlations between the harmonized factor scores and the original full-item models were high for the p-factor (>0.89) and small to moderate (0.12-0.81) for the specific factors. 6.3%-50.9% of participants presented factor score differences between harmonized and original models higher than 0.5 z-score.

CONCLUSIONS

The CBCL-GOASSESS harmonization indicates that few models provide reliable specific factors and are instrument-invariant. Moreover, authenticity was high for the p-factor and moderate for specific factors. Future studies can use this framework to examine the impact of harmonizing instruments in psychiatric research.

Articulating biological and social approaches in child and adolescent psychiatry

Child and adolescent psychiatry has been based on numerous fields of research and theories, including neuroscience, physiology, psychology (developmental, psychodynamic, systemic, cognitive-behavioral, etc.), anthropology, sociology, and education sciences. Integrating transdisciplinary knowledge in multi-level models is an ongoing challenge for the future that is not immediately applicable in clinical practice and research. Articulating, i.e., to connect, to be jointed, (psycho)biological and (psycho)social approaches in child and adolescent psychiatry is a daily challenge for clinicians and researchers. Research is often limited to specific fields whereas real-life clinical practice needs a pluralistic approach. Research designs, tools, and clinical training need to provide knowledge applicable to the necessarily pluralistic daily clinical practice. This article provides some perspectives on how to articulate biological and social approaches, from research to clinical practice, and discusses the concept of pluralistic approaches, multimodal interventions, and how to provide articulated mental health care and training. Suggestions to better articulate biological and social approaches are provided: (I) State that the research object can be approached from different theoretical, research and clinical angles and explain the one chosen; (II) Propose synthesis articles that articulate biological and social knowledge; (III) Design biological studies that take into account social factors, and design social studies that take into account biological factors; (IV) Design transcultural tools; (V) Build pluralistic interventions, i.e., therapeutic modalities and mental health care settings that articulate biological and social approaches; (VII) Develop training in pluralistic articulated care.

Assessing Measurement Invariance Across Multiple Groups: When Is Fit Good Enough?

Complex research questions often need large samples to obtain accurate estimates of parameters and adequate power. Combining extant data sets into a large, pooled data set is one way this can be accomplished without expending resources. Measurement invariance (MI) modeling is an established approach to ensure participant scores are on the same scale. There are two major problems when combining independent data sets through MI. First, sample sizes will often be large leading to small differences becoming noninvariant. Second, not all data sets may include the same combination of measures. In this article, we present a method that can deal with both these problems and is user friendly. It is a combination of generating random normal deviates for variables missing completely in combination with assessing model fit using the root mean square error of approximation good enough principle, based on the hypothesis that the difference between groups is not zero but small. We demonstrate the method by examining MI across eight independent data sets and compare the MI decisions of the traditional and good enough approach. Our results show the approach has potential in combining educational data.

Rationale and Design of the National Neuropsychology Network

OBJECTIVE

The National Neuropsychology Network (NNN) is a multicenter clinical research initiative funded by the National Institute of Mental Health (NIMH; R01 MH118514) to facilitate neuropsychology's transition to contemporary psychometric assessment methods with resultant improvement in test validation and assessment efficiency.

METHOD

The NNN includes four clinical research sites (Emory University; Medical College of Wisconsin; University of California, Los Angeles (UCLA); University of Florida) and Pearson Clinical Assessment. Pearson Q-interactive (Q-i) is used for data capture for Pearson published tests; web-based data capture tools programmed by UCLA, which serves as the Coordinating Center, are employed for remaining measures.

RESULTS

NNN is acquiring item-level data from 500-10,000 patients across 47 widely used Neuropsychology (NP) tests and sharing these data via the NIMH Data Archive. Modern psychometric methods (e.g., item response theory) will specify the constructs measured by different tests and determine their positive/negative predictive power regarding diagnostic outcomes and relationships to other clinical, historical, and demographic factors. The Structured History Protocol for NP (SHiP-NP) helps standardize acquisition of relevant history and self-report data.

CONCLUSIONS

NNN is a proof-of-principle collaboration: by addressing logistical challenges, NNN aims to engage other clinics to create a national and ultimately an international network. The mature NNN will provide mechanisms for data aggregation enabling shared analysis and collaborative research. NNN promises ultimately to enable robust diagnostic inferences about neuropsychological test patterns and to promote the validation of novel adaptive assessment strategies that will be more efficient, more precise, and more sensitive to clinical contexts and individual/cultural differences.

Lp Loss Functions in Invariance Alignment and Haberman Linking with Few or Many Groups

The comparison of group means in latent variable models plays a vital role in empirical research in the social sciences. The present article discusses an extension of invariance alignment and Haberman linking by choosing the robust power loss function ρ(x)=|x|p(p>0). This power loss function with power values p smaller than one is particularly suited for item responses that are generated under partial invariance. For a general class of linking functions, asymptotic normality of estimates is shown. Moreover, the theory of M-estimation is applied for obtaining linking errors (i.e., inference with respect to a population of items) for this class of linking functions. In a simulation study, it is shown that invariance alignment and Haberman linking have comparable performance, and in some conditions, the newly proposed robust Haberman linking outperforms invariance alignment. In three examples, the influence of the choice of a particular linking function on the estimation of group means is demonstrated. It is concluded that the choice of the loss function in linking is related to structural assumptions about the pattern of noninvariance in item parameters.