Studying Individual Differences in Predictability With Gamma Regression and Nonlinear Multilevel Models

Catching Up on Multilevel Modeling

This review focuses on the use of multilevel models in psychology and other social sciences. We target readers who are catching up on current best practices and sources of controversy in the specification of multilevel models. We first describe common use cases for clustered, longitudinal, and cross-classified designs, as well as their combinations. Using examples from both clustered and longitudinal designs, we then address issues of centering for observed predictor variables: its use in creating interpretable fixed and random effects of predictors, its relationship to endogeneity problems (correlations between predictors and model error terms), and its translation into multivariate multilevel models (using latent-centering within multilevel structural equation models). Finally, we describe novel extensions—mixed-effects location–scale models—designed for predicting differential amounts of variability.

Association between area- and individual-level socio-economic factors with glycated haemoglobin-Evidence from a Portuguese population-based study

AIMS

This study aims to estimate the associations between area-level deprivation and individual-level socio-economic factors, as well as their interaction, with glycated haemoglobin (HbA_1c ) levels.

METHODS

We conducted a gamma multilevel regression analysis using individual-level data from the Portuguese National Health Examination Survey and a deprivation index built through factor analysis, at municipality level, with census variables.

RESULTS

Living in a municipality with high material deprivation and having a low level of education were independently associated with an increase of 2.3% (95% confidence interval [CI] 0.6, 4.0) and of 1.6% (95% CI 0.6, 2.7) in the mean levels of HbA_1c , respectively. The interaction between area material deprivation and individual-level education was not associated with the levels of HbA_1c (0.5%, 95% CI -1.3, 2.3).

CONCLUSIONS

Our findings support the collective resources model that argues that people in less deprived areas have better health because there are more collective resources. The results suggest that to reduce socio-economic inequalities associated with the levels of HbA_1c and, consequently, with diabetes, will require attention to the area material deprivation and individual-level education. Upstream social determinants of health are thus highlighted.

From Nuisance to Novel Research Questions: Using Multilevel Models to Predict Heterogeneous Variances

Constructs that reflect differences in variability are of interest to many researchers studying workplace phenomena. The aggregation methods typically used to investigate “variability-based” constructs suffer from several limitations, including the inability to include Level 1 predictors and a failure to account for uncertainty in the variability estimates. We demonstrate how mixed-effects location-scale (MELS) and heterogeneous variance models, which are direct extensions of traditional mixed-effects (or multilevel) models, can be used to test mean (location)- and variability (scale)-related hypotheses simultaneously. The aims of this article are to demonstrate (a) how the MELS and heterogeneous variance models can be estimated with both nested cross-sectional and longitudinal data to answer novel research questions about constructs of interest to organizational researchers, (b) how a Bayesian approach allows for the inclusion of random intercepts and slopes when predicting both variability and mean levels, and finally (c) how researchers can use a multilevel approach to predict between-group heterogeneous variances. In doing so, this article highlights the added value of viewing variability as more than a statistical nuisance in organizational research.

Detecting Consensus Emergence in Organizational Multilevel Data: Power Simulations

Theories suggest that groups within organizations often develop shared values, beliefs, affect, behaviors, or agreed-on routines; however, researchers rarely study predictors of consensus emergence over time. Recently, a multilevel-methods approach for detecting and studying emergence in organizational field data has been described. This approach—the consensus emergence model—builds on an extended three-level multilevel model. Researchers planning future studies based on the consensus emergence model need to consider (a) sample size characteristics required to detect emergence effects with satisfactory statistical power and (b) how the distribution of the overall sample size across the levels of the multilevel model influences power. We systematically address both issues by conducting a power simulation for detecting main and moderating effects involving consensus emergence under a variety of typical research scenarios and provide an R-based tool that readers can use to estimate power. Our discussion focuses on the future use and development of multilevel methods for studying emergence in organizational research.

High-Stakes Testing Case Study: A Latent Variable Approach for Assessing Measurement and Prediction Invariance

The existence of differences in prediction systems involving test scores across demographic groups continues to be a thorny and unresolved scientific, professional, and societal concern. Our case study uses a two-stage least squares (2SLS) estimator to jointly assess measurement invariance and prediction invariance in high-stakes testing. So, we examined differences across groups based on latent as opposed to observed scores with data for 176 colleges and universities from The College Board. Results showed that evidence regarding measurement invariance was rejected for the SAT mathematics (SAT-M) subtest at the 0.01 level for 74.5% and 29.9% of cohorts for Black versus White and Hispanic versus White comparisons, respectively. Also, on average, Black students with the same standing on a common factor had observed SAT-M scores that were nearly a third of a standard deviation lower than for comparable Whites. We also found evidence that group differences in SAT-M measurement intercepts may partly explain the well-known finding of observed differences in prediction intercepts. Additionally, results provided evidence that nearly a quarter of the statistically significant observed intercept differences were not statistically significant at the 0.05 level once predictor measurement error was accounted for using the 2SLS procedure. Our joint measurement and prediction invariance approach based on latent scores opens the door to a new high-stakes testing research agenda whose goal is to not simply assess whether observed group-based differences exist and the size and direction of such differences. Rather, the goal of this research agenda is to assess the causal chain starting with underlying theoretical mechanisms (e.g., contextual factors, differences in latent predictor scores) that affect the size and direction of any observed differences.

An Improved Correction for Range Restricted Correlations Under Extreme, Monotonic Quadratic Nonlinearity and Heteroscedasticity

Standardized tests are frequently used for selection decisions, and the validation of test scores remains an important area of research. This paper builds upon prior literature about the effect of nonlinearity and heteroscedasticity on the accuracy of standard formulas for correcting correlations in restricted samples. Existing formulas for direct range restriction require three assumptions: (1) the criterion variable is missing at random; (2) a linear relationship between independent and dependent variables; and (3) constant error variance or homoscedasticity. The results in this paper demonstrate that the standard approach for correcting restricted correlations is severely biased in cases of extreme monotone quadratic nonlinearity and heteroscedasticity. This paper offers at least three significant contributions to the existing literature. First, a method from the econometrics literature is adapted to provide more accurate estimates of unrestricted correlations. Second, derivations establish bounds on the degree of bias attributed to quadratic functions under the assumption of a monotonic relationship between test scores and criterion measurements. New results are presented on the bias associated with using the standard range restriction correction formula, and the results show that the standard correction formula yields estimates of unrestricted correlations that deviate by as much as 0.2 for high to moderate selectivity. Third, Monte Carlo simulation results demonstrate that the new procedure for correcting restricted correlations provides more accurate estimates in the presence of quadratic and heteroscedastic test score and criterion relationships.

An IRT Modeling Approach for Assessing Item and Person Discrimination in Binary Personality Responses

Conventional item response theory (IRT) modeling of personality responses considers two item characteristics-location and discrimination-but only one person characteristic-location or trait level. An IRT modeling approach that jointly considers item and person discriminations, however, is thought to be more realistic and appropriate in this domain and has several potential advantages. This article develops a model of this type for unidimensional binary responses together with procedures for estimating item and person parameters and assessing model appropriateness (including person fit). A series of preliminary simulations suggests that the approach is feasible, and a real-data example illustrates the potential advantages with respect to the standard two-parameter model. Limitations of the proposal and further work are also discussed.

A Factor-Analytic Model for Assessing Individual Differences in Response Scale Usage

This article proposes a factor-analytic model, intended for graded-response or continuous-response personality and attitude items, which includes an additional multiplicative person parameter that models the individual's response mapping process. The model, which is a modification of Spearman's (1904) factor analysis (FA) model, is parameterized as both an FA model and an item response theory (IRT) model and is fully developed to the extent that it can be used in applications. Procedures for (a) calibrating the items and assessing data fit, (b) obtaining individual estimates of both person parameters, (c) determining measurement precision, and (d) assessing differential predictability are proposed and discussed. The potential advantages of the proposal, its practical relevance, and its relations with other approaches are also discussed. Its functioning is assessed with a simulation study and 3 empirical examples in the personality domain.

Young women's dynamic family size preferences in the context of transitioning fertility

Dynamic theories of family size preferences posit that they are not a fixed and stable goal but rather are akin to a moving target that changes within individuals over time. Nonetheless, in high-fertility contexts, changes in family size preferences tend to be attributed to low construct validity and measurement error instead of genuine revisions in preferences. To address the appropriateness of this incongruity, the present study examines evidence for the sequential model of fertility among a sample of young Malawian women living in a context of transitioning fertility. Using eight waves of closely spaced data and fixed-effects models, we find that these women frequently change their reported family size preferences and that these changes are often associated with changes in their relationship and reproductive circumstances. The predictability of change gives credence to the argument that ideal family size is a meaningful construct, even in this higher-fertility setting. Changes are not equally predictable across all women, however, and gamma regression results demonstrate that women for whom reproduction is a more distant goal change their fertility preferences in less-predictable ways.

Using the Criterion-Predictor Factor Model to Compute the Probability of Detecting Prediction Bias with Ordinary Least Squares Regression

The study of prediction bias is important and the last five decades include research studies that examined whether test scores differentially predict academic or employment performance. Previous studies used ordinary least squares (OLS) to assess whether groups differ in intercepts and slopes. This study shows that OLS yields inaccurate inferences for prediction bias hypotheses. This paper builds upon the criterion-predictor factor model by demonstrating the effect of selection, measurement error, and measurement bias on prediction bias studies that use OLS. The range restricted, criterion-predictor factor model is used to compute Type I error and power rates associated with using regression to assess prediction bias hypotheses. In short, OLS is not capable of testing hypotheses about group differences in latent intercepts and slopes. Additionally, a theorem is presented which shows that researchers should not employ hierarchical regression to assess intercept differences with selected samples.