Improvement of Norm Score Quality via Regression-Based Continuous Norming

Montreal Cognitive Assessment's auditory items (MoCA-22): Normative data and reliable change indices

Our objective was to establish normative data and reliable change indices (RCI) for the Montreal Cognitive Assessment's auditory items (MoCA-22). 4,935 cognitively unimpaired participants were administered the MoCA during an in-person visit to an Alzheimer's Disease Research Center (Mage = 67.9, Meducation = 16.2, 65.8% women, 75.9% non-Hispanic-White), with 2,319 unimpaired participants returning for follow-up. Normative values and cutoffs were developed using demographic predictions from ordinary and quantile regression. Test-retest reliability was calculated using Spearman and intraclass correlations. RCI values were calculated using Chelune and colleagues' (1993) formula. Education, age, and sex were all statistically related to MoCA-22 scores, with education having the strongest relationship. Notably, these relationships were not consistent across MoCA-22 quantiles, with education becoming more important and sex becoming less important for predicting low scores. These models were integrated into a calculator for deriving normative scores for an individual case. Furthermore, there was adequate-to-good test-retest reliability (ϱ = 0.56 95% CI [.54, .59]; ICC = 0.75, 95% CI [.73, .77]) and changes of at least 2-3 points are necessary to identify reliable change at 1-3-year follow-up. These findings add to the literature regarding utility of the MoCA-22 in the cognitive screening of older adults.

A tutorial on automatic post-stratification and weighting in conventional and regression-based norming of psychometric tests

Norm scores are an essential source of information in individual diagnostics. Given the scope of the decisions this information may entail, establishing high-quality, representative norms is of tremendous importance in test construction. Representativeness is difficult to establish, though, especially with limited resources and when multiple stratification variables and their joint probabilities come into play. Sample stratification requires knowing which stratum an individual belongs to prior to data collection, but the required variables for the individual's classification, such as socio-economic status or demographic characteristics, are often collected within the survey or test data. Therefore, post-stratification techniques, like iterative proportional fitting (= raking), aim at simulating representativeness of normative samples and can thus enhance the overall quality of the norm scores. This tutorial describes the application of raking to normative samples, the calculation of weights, the application of these weights in percentile estimation, and the retrieval of continuous, regression-based norm models with the cNORM package on the R platform. We demonstrate this procedure using a large, non-representative dataset of vocabulary development in childhood and adolescence (N = 4542), using sex and ethnical background as stratification variables.

Continuous Norming Approaches: A Systematic Review and Real Data Example

Norming of psychological tests is decisive for test score interpretation. However, conventional norming based on subgroups results either in biases or require very large samples to gather precise norms. Continuous norming methods, namely inferential, semi-parametric, and (simplified) parametric norming, propose to solve those issues. This article provides a systematic review of continuous norming. The review includes 121 publications with overall 189 studies. The main findings indicate that most studies used simplified parametric norming, not all studies considered essential distributional assumptions, and the evidence comparing different norming methods is inconclusive. In a real data example, using the standardization sample of the Need for Cognition-KIDS scale, we compared the precision of conventional, semi-parametric, and parametric norms. A hierarchy in terms of precision emerged with conventional norms being least precise, followed by semi-parametric norms, and parametric norms being most precise. We discuss these findings by comparing our findings and methods to previous studies.

Age-stratified norms for Raven's standard progressive matrices for Sri Lankan adults

OBJECTIVE

The aim of this study was to create age-stratified norms for the Raven's Standard Progressive Matrices (SPM) for Sri Lankan adults.

METHODS

A sample of 610 adults (age: 18-72 years; education: 1-19 years), underwent the 60-item version of the SPM under individual supervision of a test administrator. The sample was stratified into 5-year age bands, and the norms are presented as percentile tables and percentile curves.

RESULTS

The age-related changes were more accurately predicted by a curvilinear model (overall R2 = 0.961) than a linear regression model (R2 = 0.639). The SPM norms are presented as age-stratified percentile tables, as well as sex-, age- and education-adjusted multiple regression equations. The highest percentiles in the younger end of the age spectrum showed a ceiling effect. In the context of age-stratified US (1993) and British (1992) norms, older individuals in the Sri Lankan sample scored much lower than their Western counterparts. However, the difference narrowed in the younger age bands, showing no difference among the 18-to-22-year age bands in the three countries.

CONCLUSIONS

This age-by-country interaction can be partly explained by poorer education in the older individuals in the present sample compared to those in the US and UK standardization samples. SPM norms presented in this paper fill a hiatus in assessment of general intellectual ability in Sri Lankan adults. Given that Sri Lanka improves its educational, socioeconomic and health standards faster than the nations who have already reached higher standards, these norms would require re-standardization in the coming decades.

Reducing the Bias of Norm Scores in Non-Representative Samples: Weighting as an Adjunct to Continuous Norming Methods

We investigated whether the accuracy of normed test scores derived from non-demographically representative samples can be improved by combining continuous norming methods with compensatory weighting of test results. To this end, we introduce Raking, a method from social sciences, to psychometrics. In a simulated reference population, we modeled a latent cognitive ability with a typical developmental gradient, along with three demographic variables that were correlated to varying degrees with the latent ability. We simulated five additional populations representing patterns of non-representativeness that might be encountered in the real world. We subsequently drew smaller normative samples from each population and used an one-parameter logistic Item Response Theory (IRT) model to generate simulated test results for each individual. Using these simulated data, we applied norming techniques, both with and without compensatory weighting. Weighting reduced the bias of the norm scores when the degree of non-representativeness was moderate, with only a small risk of generating new biases.

Norms for the Triana Test: A Story Recall Test Based on Emotional Material

Background

The "Triana Test" is a novel story recall test based on emotional material with demonstrated accuracy in diagnosing mild cognitive impairment patients.

Objective

This study aims to obtain normative data for the "Triana Test".

Methods

A normative study was conducted at a university hospital in Spain. Partners of patients were systematically recruited if eligible (age ≥50, no memory complaints, and a total TMA-93 score at or above the 10th percentile). The "Triana Test" was administered and scored. For developing the normative data, a regression-based method was followed.

Results

The final sample included 362 participants (median age = 66, range = 50-88; 64.9% females). A model including age and educational level better predicted the total scores. Combinations of these variables resulted in different 10th percentile scores.

Conclusions

Norms for using the "Triana Test" are now available. The provided cutoffs for the 10th percentile will aid in the diagnosis of prodromal Alzheimer's disease.

The impact of conventional versus robust norming on cognitive characterization and clinical classification of MCI and dementia

We examined the impact of conventional versus robust normative approaches on cognitive characterization and clinical classification of MCI versus dementia. The sample included participants from the National Alzheimer's Coordinating Center Uniform Data Set. Separate demographically adjusted z-scores for cognitive tests were derived from conventional (n = 4273) and robust (n = 602) normative groups. To assess the impact of deriving scores from a conventional versus robust normative group on cognitive characterization, we examined likelihood of having a low score on each neuropsychological test. Next, we created receiver operating characteristic (ROC) curves for the ability of normed scores derived from each normative group to differentiate between MCI (n = 3570) and dementia (n = 1564). We examined the impact of choice of normative group on classification accuracy by comparing sensitivity and specificity values and areas under the curves (AUC). Compared with using a conventional normative group, using a robust normative group resulted in a higher likelihood of low cognitive scores for individuals classified with MCI and dementia. Comparison of the classification accuracy for distinguishing MCI from dementia did not suggest a statistically significant advantage for either normative approach (Z = -0.29, p = .77; AUC = 0.86 for conventional and AUC = 0.86 for robust). In summary, these results indicate that using a robust normative group increases the likelihood of characterizing cognitive performance as low. However, there is not a clear advantage of using a robust over a conventional normative group when differentiating between MCI and dementia.

Normative Data in Cross-Cultural Neuropsychological Instruments in Spaniards and Colombians Living in Spain

BACKGROUND

Cross-Cultural Dementia Screening (CCD), Rowland Universal Dementia Assessment Scale (RUDAS), and European Cross-cultural Neuropsychological Test Battery (CNTB) are three novel neuropsychological instruments developed from a cross-cultural perspective to reduce the impact of culture in cognitive assessment and improve the assessment in diverse populations.

OBJECTIVE

We aimed to collect and present normative data on these tests in a majority population sample (Spaniards living in Spain) and in a minority population sample (Colombians living in Spain).

METHODS

CCD, RUDAS, and CNTB were administered to a group of 300 cognitively healthy participants (150 Spaniards and 150 Colombians). Linear regression modeling strategy was used to provide adjusted norms for demographic factors and to explore the influence of these factors on test performance.

RESULTS

Most of the CCD and CNTB scores were predicted by age and years of education, with some tests only predicted by age or showing a ceiling effect. The comparison of normative data between the two samples confirmed the favorable cross-cultural properties of these instruments, with only some differences in processing speed and executive functioning scores.

CONCLUSIONS

Our study finds a comparable influence of demographic factors in both populations on the performance of CCD, RUDAS, and CNTB, confirming their adequate cross-cultural properties. We provide normative data for these tests in Spaniards and Colombians living in Spain.

Students' learning growth in mental addition and subtraction: Results from a learning progress monitoring approach

The purpose of this study was to measure and describe students' learning development in mental computation of mixed addition and subtraction tasks up to 100. We used a learning progress monitoring (LPM) approach with multiple repeated measurements to examine the learning curves of second-and third-grade primary school students in mental computation over a period of 17 biweekly measurement intervals in the school year 2020/2021. Moreover, we investigated how homogeneous students' learning curves were and how sociodemographic variables (gender, grade level, the assignment of special educational needs) affected students' learning growth. Therefore, 348 German students from six schools and 20 classes (10.9% students with special educational needs) worked on systematically, but randomly mixed addition and subtraction tasks at regular intervals with an online LPM tool. We collected learning progress data for 12 measurement intervals during the survey period that was impacted by the COVID-19 pandemic. Technical results show that the employed LPM tool for mental computation met the criteria of LPM research stages 1 and 2. Focusing on the learning curves, results from latent growth curve modeling showed significant differences in the intercept and in the slope based on the background variables. The results illustrate that one-size-fits-all instruction is not appropriate, thus highlighting the value of LPM or other means that allow individualized, adaptive teaching. The study provides a first quantitative overview over the learning curves for mental computation in second and third grade. Furthermore, it offers a validated tool for the empirical analysis of learning curves regarding mental computation and strong reference data against which individual learning growth can be compared to identify students with unfavorable learning curves and provide targeted support as part of an adaptive, evidence-based teaching approach. Implications for further research and school practice are discussed.

Common measures or common metrics? A plea to harmonize measurement results

OBJECTIVE

There is a great variety of measurement instruments to assess similar constructs in clinical research and practice. This complicates the interpretation of test results and hampers the implementation of measurement-based care.

METHOD

For reporting and discussing test results with patients, we suggest converting test results into universally applicable common metrics. Two well-established metrics are reviewed: T scores and percentile ranks. Their calculation is explained, their merits and drawbacks are discussed, and recommendations for the most convenient reference group are provided.

RESULTS

We propose to express test results as T scores with the general population as reference group. To elucidate test results to patients, T scores may be supplemented with percentile ranks, based on data from a clinical sample. The practical benefits are demonstrated using the published data of four frequently used instruments for measuring depression: the CES-D, PHQ-9, BDI-II and the PROMIS depression measure.

DISCUSSION

Recent initiatives have proposed to mandate a limited set of outcome measures to harmonize clinical measurement. However, the selected instruments are not without flaws and, potentially, this directive may hamper future instrument development. We recommend using common metrics as an alternative approach to harmonize test results in clinical practice, as this will facilitate the integration of measures in day-to-day practice.

Modelling Norm Scores with the cNORM Package in R

In this article, we explain and demonstrate how to model norm scores with the cNORM package in R. This package is designed specifically to determine norm scores when the latent ability to be measured covaries with age or other explanatory variables such as grade level. The mathematical method used in this package draws on polynomial regression to model a three-dimensional hyperplane that smoothly and continuously captures the relation between raw scores, norm scores and the explanatory variable. By doing so, it overcomes the typical problems of classical norming methods, such as overly large age intervals, missing norm scores, large amounts of sampling error in the subsamples or huge requirements with regard to the sample size. After a brief introduction to the mathematics of the model, we describe the individual methods of the package. We close the article with a practical example using data from a real reading comprehension test.

In norming we trust

Zusammenfassung. Ziel der Untersuchung war ein systematischer Vergleich verschiedener Verfahren zur Normdatenmodellierung. Der auf Taylor-Polynomen basierende semi-parametrische Normierungsansatz (SPCN) mittels cNORM ( Lenhard, Lenhard & Gary, 2018 ) wurde parametrischen Anpassungen basierend auf Generalized Additive Models for Location, Scale and Shape (GAMLSS; Stasinopoulos et al., 2018 ) gegenübergestellt und die Normierungsgüte in Abhängigkeit der Faktoren Normstichprobengröße ( n = 525, 700, 1 050, 1 750), Itemanzahl (i = 10, 20, 40) sowie Itemschwierigkeit analysiert. Die Modellierung erfolgte kreuzvalidiert auf der Basis simulierter Rohdaten von Normierungs- und Validierungsstichproben: Mittels der verschiedenen Verfahren wurden auf der Basis der Normierungsstichprobe statistische Modelle berechnet und auf die Validierungsstichprobe übertragen, um die jeweils vorhergesagten mit den tatsächlichen Normwerten zu vergleichen. Der semi-parametrische Ansatz lieferte in den meisten Fällen den geringsten Normierungsfehler und damit das beste Normierungsergebnis. Die deutlichsten Unterschiede fanden sich bei leichten bzw. schweren Testskalen in Verbindung mit einer kleinen Itemanzahl. Der Einfluss der Normstichprobengröße war bei allen Methoden vergleichbar.