Using Person Fit Statistics to Detect Outliers in Survey Research

The Oxford Agoraphobic Avoidance Scale

BACKGROUND

Agoraphobic avoidance of everyday situations is a common feature in many mental health disorders. Avoidance can be due to a variety of fears, including concerns about negative social evaluation, panicking, and harm from others. The result is inactivity and isolation. Behavioural avoidance tasks (BATs) provide an objective assessment of avoidance and in situ anxiety but are challenging to administer and lack standardisation. Our aim was to draw on the principles of BATs to develop a self-report measure of agoraphobia symptoms.

METHOD

The scale was developed with 194 patients with agoraphobia in the context of psychosis, 427 individuals in the general population with high levels of agoraphobia, and 1094 individuals with low levels of agoraphobia. Factor analysis, item response theory, and receiver operating characteristic analyses were used. Validity was assessed against a BAT, actigraphy data, and an existing agoraphobia measure. Test-retest reliability was assessed with 264 participants.

RESULTS

An eight-item questionnaire with avoidance and distress response scales was developed. The avoidance and distress scales each had an excellent model fit and reliably assessed agoraphobic symptoms across the severity spectrum. All items were highly discriminative (avoidance: a = 1.24-5.43; distress: a = 1.60-5.48), indicating that small increases in agoraphobic symptoms led to a high probability of item endorsement. The scale demonstrated good internal reliability, test-retest reliability, and validity.

CONCLUSIONS

The Oxford Agoraphobic Avoidance Scale has excellent psychometric properties. Clinical cut-offs and score ranges are provided. This precise assessment tool may help focus attention on the clinically important problem of agoraphobic avoidance.

Psychometric properties of the generalized anxiety disorder-7 (GAD-7) in a sample of workers

Objective

To evaluate the psychometric properties of the GAD-7 by obtaining evidence of internal structure (dimensionality, precision and differential functioning of items) and association with external variables.

Methods

A total of 2,219 protocols from three different studies conducted with Puerto Rican employees that administered the GAD-7 were selected for the current study. Item response theory modeling was used to assess internal structure, and linear association with external variables.

Results

The items were adapted to a graduated response model, with high similarity in the discrimination and location parameters, as well as in the precision at the level of the items and in the total score. No violation of local independence and differential item functioning was detected. The association with convergent (work-related rumination) and divergent (work engagement, sex, and age) variables were theoretically consistent.

Conclusion

The GAD-7 is a psychometrically robust tool for detecting individual variability in symptoms of anxiety in workers.

Development, validation and item reduction of a food literacy questionnaire (IFLQ-19) with Australian adults

Background

Food literacy is theorised to improve diet quality, nutrition behaviours, social connectedness and food security. The definition and conceptualisation by Vidgen & Gallegos, consisting of 11 theoretical components within the four domains of planning and managing, selecting, preparing and eating, is currently the most highly cited framework. However, a valid and reliable questionnaire is needed to comprehensively measure this conceptualisation. Therefore, this study draws on existing item pools to develop a comprehensive food literacy questionnaire using item response theory.

Methods

Five hundred Australian adults were recruited in Study 1 to refine a food literacy item pool using principal component analysis (PCA) and item response theory (IRT) which involved detailed item analysis on targeting, responsiveness, validity and reliability. Another 500 participants were recruited in Study 2 to replicate item analysis on validity and reliability on the refined item pool, and 250 of these participants re-completed the food literacy questionnaire to determine its test–retest reliability.

Results

The PCA saw the 171-item pool reduced to 100-items across 19 statistical components of food literacy. After the thresholds of 26 items were combined, responses to the food literacy questionnaire had ordered thresholds (targeting), acceptable item locations (< -0.01 to + 1.53) and appropriateness of the measurement model (n = 92% expected responses) (responsiveness), met outfit mean-squares MSQ (0.48—1.42) (validity) and had high person, item separation (> 0.99) and test–retest (ICC 2,1 0.55–0.88) scores (reliability).

Conclusions

We developed a 100-item food literacy questionnaire, the IFLQ-19 to comprehensively address the Vidgen & Gallegos theoretical domains and components with good targeting, responsiveness, reliability and validity in a diverse sample of Australian adults.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12966-022-01351-8.

Satisfaction towards virtual courses: Development and validation of a short measure in COVID-19 times

This study was aimed to develop and validate a short scale to measure satisfaction with virtual courses (SVC-S) in a sample of higher education students during the covid-19 pandemic; specifically, in the year 2021. A total of 3080 students between 16 and 56 years of age participated (Mean = 25.71; SD = 8.83); 1836 were female (59.60 %) and 1244 male (40.40 %). The participants were students from three cities in Peru (77.90% from Lima, 12.70% from Trujillo and 9.42% from Cajamarca). Qualitative and quantitative procedures were followed for the construction of the SVC-S. Item response theory (IRT) considering Samejima's two-parameter Graded Response Model (GRM) (2PL) and the test-item information function was used to establish accuracy/reliability, and the relationship of the SVC-S with a similar measure was examined to demonstrate convergence and discrimination. The results reveal that the data present an optimal fit (M2 (2) = 3.62; RMSEA = .016; CFI = 1.00). Reliability is excellent (r _xx = .93) and the information function suggests that the instrument is more accurate at low levels of the latent trait. Regarding convergence with an academic satisfaction scale, the SVC-S showed an appropriate correlation (r = .70) whose average variance extracted (AVE) reported good discrimination of the constructs; despite being conceptually similar. SVC-S is concluded to be a valid and reliable measure that can be used in future studies in higher education.

Do You Recall?: Results From a Within-Person Recall Study of the Patient-Reported Outcomes Measurement Information System (PROMIS) Short Form v2.0 - Physical Function 8c

OBJECTIVES

This study aimed to determine whether responses to Patient-Reported Outcomes Measurement Information System Short Form v2.0 - Physical Function 8c (PROMIS PF8c) items differed when the use of a 7-day recall period was compared with no specified recall period.

METHODS

Using a within-subject design, we surveyed 1810 individuals from the US general population, administering PROMIS PF8c at survey beginning and end. The order of measure presentation was randomly assigned. We calculated recall difference scores (RDSs) as no recall score minus 7-day recall score using both item response theory-based T scores and raw summed scores. We examined the distribution and created Bland-Altman plots for both RDS_Tscore and RDS_Raw. We also calculated correlations between no recall versus 7-day recall T score and raw scores. Finally, we determined whether differences in no recall versus 7-day recall scores were associated with patient-reported PF.

RESULTS

RDS_Tscore and RDS_Raw had means (root mean square differences) of 0.00 (5.43) and -0.04 (3.79), respectively. The vast majority (%) of RDS_Tscore and RDS_Raw values fell between the Bland-Altman limits of agreement (-10.65 to 10.66 and -7.46 to 7.38, respectively). Pearson's correlations between no recall and 7-day recall for T scores and raw scores were 0.88 and 0.87, respectively. Effect sizes for mean RDS_Tscore and RDS_Raw compared across level of Eastern Oncology Cooperative Group performance status, patient global impression of PF severity, and single PF items were near 0.

CONCLUSIONS

We did not find any significant recall period effect on PF8c responses. Therefore, we recommend the use of the PROMIS physical function standard, with no specified recall time period.

Reliability and Validity of the Spanish Version of the Medical Outcomes Study Pain Severity Scale in Mexican Patients With Rheumatic Diseases

OBJECTIVE

To evaluate the reliability and validity of the Spanish version of the Medical Outcomes Study Pain Severity Scale (MOS-PSS) in patients with rheumatic diseases.

METHODS

A cross-sectional study was carried out from January 2019 to March 2020 using a nonprobabilistic sampling of consecutive cases. The internal consistency of the scale was evaluated through the Cronbach α coefficient and by the item response theory reliability coefficient. Construct validity was evaluated by factor analysis; relationships between MOS-PSS and visual numeric scale for pain, numeric rating scale for pain, and verbal rating scale for pain; and differences between relevant groups. Item response theory-based methods were used to assess item performance.

RESULTS

A clinical sample of 796 outpatients was recruited. Most patients presented moderate to severe pain. Two subscale solutions showed a good model fit in confirmatory factor analysis. The overall model fit of multidimensional generalized partial credit model showed to be adequate. The most discriminating item was "average pain intensity." Evidence revealed disordered thresholds in 2 items. Collapsing categories resulted in ordered thresholds for all items and significantly improving the overall model fit. The MOS-PSS and modified MOS-PSS yielded high reliability. Both scales were very strongly correlated with numeric rating scale for pain, visual numeric scale for pain, and verbal rating scale for pain (ρ ≥ 0.85). All hypotheses related to subgroups comparison were fulfilled.

CONCLUSIONS

Overall, the Spanish version of the MOS-PSS showed good reliability and construct validity. Nevertheless, the statistical evidence from this study would suggest the modified MOS-PSS should be the version of choice for measuring pain in Mexican patients with rheumatic diseases.

Mental health and its association with coping strategies and intolerance of uncertainty during the COVID-19 pandemic among the general population in Saudi Arabia: cross-sectional study

BACKGROUND

The COVID-19 pandemic has had a major impact on people's lives globally. The outbreak in Saudi Arabia worsened when the number of cases and deaths rose in March and April of 2020, leading to a national lockdown. This study aimed to assess the factors associated with mental health symptoms in a sample of people residing in Saudi Arabia during the COVID-19 pandemic.

METHODS

We conducted an observational cross-sectional study using an online survey distributed via social media, completed by 3032 respondents from all Saudi regions. We collected demographic data, illness history, and scores of validated self-report scales to assess mental health symptoms, intolerance of uncertainty, and coping strategies.

RESULTS

In total, respondents indicated moderate to very severe symptoms during the pandemic as follows: 20.9% for depression, 17.5% for anxiety, and 12.6% for stress. Younger age, female gender, and history of mental illness were associated with higher levels of depression, anxiety, stress, and insomnia. Intolerance of uncertainty and certain coping strategies (such as denial or self-blame) were associated with more severe symptoms.

CONCLUSIONS

Mental health is a key concern during the COVID-19 pandemic, especially for the identified vulnerable groups. Agencies concerned with mental health during crises may use the studied associated factors of mental health symptoms to generate targeted policies or interventions.

Validation of a Spanish version of the Health Assessment Questionnaire-II to assess Mexican patients' physical function with rheumatoid arthritis

OBJECTIVE

To evaluate the validity, reliability, and performance of the Health Assessment Questionnaire-II (HAQ-II) Spanish version questionnaire to measure physical function.

METHODS

A cross-sectional study of 496 patients with rheumatoid arthritis (RA), distributed in 2 samples. The construct validity was evaluated employing the confirmatory factor analysis (CFA) and the validity based on the relationship with other variables. Cronbach's alpha (α) and McDonald's omega (ω) coefficient were used to determine reliability. Item performance was analysed by fitting different models of item response theory (IRT).

RESULTS

The one-factor model presented a poor fit in the CFA; an exploratory factor analysis (AFE) was carried out, which suggested a 2-factor structure. The CFA in the second sample confirmed that the second-order model had a good fit to the data. The general factor explained more than 70% of the variance. The reliability indices showed adequate internal consistency (α = .92-.95; ω = .88-.93). Ninety-three percent of the contrasting hypotheses about the relationship of the HAQ-II scores with other variables were confirmed, demonstrating their convergent, divergent, and known group validity. The multidimensional graduated response model was the one that best predicted person's interaction with the items.

CONCLUSION

The Spanish version of the HAQ-II presents adequate validity and reliability for measuring Mexican patients' physical function with RA.

Framework for the Classification of Emotions in People With Visual Disabilities Through Brain Signals

Nowadays, the recognition of emotions in people with sensory disabilities still represents a challenge due to the difficulty of generalizing and modeling the set of brain signals. In recent years, the technology that has been used to study a person's behavior and emotions based on brain signals is the brain-computer interface (BCI). Although previous works have already proposed the classification of emotions in people with sensory disabilities using machine learning techniques, a model of recognition of emotions in people with visual disabilities has not yet been evaluated. Consequently, in this work, the authors present a twofold framework focused on people with visual disabilities. Firstly, auditory stimuli have been used, and a component of acquisition and extraction of brain signals has been defined. Secondly, analysis techniques for the modeling of emotions have been developed, and machine learning models for the classification of emotions have been defined. Based on the results, the algorithm with the best performance in the validation is random forest (RF), with an accuracy of 85 and 88% in the classification for negative and positive emotions, respectively. According to the results, the framework is able to classify positive and negative emotions, but the experimentation performed also shows that the framework performance depends on the number of features in the dataset and the quality of the Electroencephalogram (EEG) signals is a determining factor.

Validation of a Spanish Version of the Health Assessment Questionnaire-II to Assess Mexican Patients' Physical Function with Rheumatoid Arthritis

OBJECTIVE

To evaluate the validity, reliability, and performance of the Health Assessment Questionnaire-II(HAQ-II) Spanish version questionnaire to measure physical function.

METHODS

A cross-sectional study of 496 patients with rheumatoid arthritis, distributed in 2 samples. The construct validity was evaluated employing the confirmatory factor analysis and the validity based on the relationship with other variables. Cronbach's alpha (α) and McDonald's omega (ω) coefficient were used to determine reliability. Item performance was analysed by fitting different models of item response theory.

RESULTS

The one-factor model presented a poor fit in the confirmatory factor analysis; an exploratory factor analysis was carried out, which suggested a 2-factor structure. The confirmatory factor analysis in the second sample confirmed that the second-order model had a good fit to the data. The general factor explained more than 70% of the variance. The reliability indices showed adequate internal consistency (α=.92-.95; ω=.88-.93). Ninety-three percent of the contrasting hypotheses about the relationship of the HAQ-II scores with other variables were confirmed, demonstrating their convergent, divergent, and known group validity. The multidimensional graduated response model was the one that best predicted person's interaction with the items.

CONCLUSION

The Spanish version of the HAQ-II presents adequate validity and reliability for measuring Mexican patients' physical function with rheumatoid arthritis.

The revised Green et al., Paranoid Thoughts Scale (R-GPTS): psychometric properties, severity ranges, and clinical cut-offs

BACKGROUND

The Green et al., Paranoid Thoughts Scale (GPTS) - comprising two 16-item scales assessing ideas of reference (Part A) and ideas of persecution (Part B) - was developed over a decade ago. Our aim was to conduct the first large-scale psychometric evaluation.

METHODS

In total, 10 551 individuals provided GPTS data. Four hundred and twenty-two patients with psychosis and 805 non-clinical individuals completed GPTS Parts A and B. An additional 1743 patients with psychosis and 7581 non-clinical individuals completed GPTS Part B. Factor analysis, item response theory, and receiver operating characteristic analyses were conducted.

RESULTS

The original two-factor structure of the GPTS had an inadequate model fit: Part A did not form a unidimensional scale and multiple items were locally dependant. A Revised-GPTS (R-GPTS) was formed, comprising eight-item ideas of reference and 10-item ideas of persecution subscales, which had an excellent model fit. All items in the new Reference (a = 2.09-3.67) and Persecution (a = 2.37-4.38) scales were strongly discriminative of shifts in paranoia and had high reliability across the spectrum of severity (a > 0.90). The R-GPTS score ranges are: average (Reference: 0-9; Persecution: 0-4); elevated (Reference: 10-15; Persecution: 5-10); moderately severe (Reference: 16-20; Persecution:11-17); severe (Reference: 21-24; Persecution: 18-27); and very severe (Reference: 25+; Persecution: 28+). Recommended cut-offs on the persecution scale are 11 to discriminate clinical levels of persecutory ideation and 18 for a likely persecutory delusion.

CONCLUSIONS

The psychometric evaluation indicated a need to improve the GPTS. The R-GPTS is a more precise measure, has excellent psychometric properties, and is recommended for future studies of paranoia.

Psychometric properties of the 12-item Knee injury and Osteoarthritis Outcome Score (KOOS-12) Spanish version for people with knee osteoarthritis

To evaluate the validity, reliability, and responsiveness to change of the 12-item Knee injury and Osteoarthritis Outcome Score (KOOS) Spanish version questionnaire. This study was based on a questionnaire validation design. A cross-sectional survey of 199 patients with knee osteoarthritis (KOA) and ten healthy controls was studied to evaluate the validity and reliability of KOOS-12. One hundred and sixteen patients were assessed for test-retest reliability, and 38 patients were included for a responsiveness assessment. Structural validity was assessed by the confirmatory factor analysis (CFA). Item response theory-based methods were used to determine the performance of the items. Internal consistency reliability was appropriate for all scales (Cronbach's alpha = 0.85-0.94). The intra-class correlation coefficient of KOOS-12 scales ranged from 0.60 to 0.71. The CFA and generalized partial credit model showed that KOOS-12 scales presented a good overall model fit. No differential item functioning was found. Convergent validity was demonstrated by strong correlations (Spearman's rho ≥ 0.70) with KOOS, International Knee Documentation Committee subjective knee evaluation form (IKDC), and Knee Intermittent and Constant Osteoarthritis Pain (ICOAP). Known-groups validity showed that KOOS-12 well discriminated subgroups of patients (radiographic severity and nutritional status). Standardized response means for KOOS-12 scales were ≥ 0.75. Changes in KOOS-12 scales had a moderate to strong correlation (Pearson's r ≥ 0.40) with the changes in the KOOS, ICOAP, and IKDC scales. The KOOS-12 Spanish version is a valid, reliable, and responsiveness to change questionnaire to measure patients' opinions about their knee and associated problems in Mexican subjects with KOA. Key Points • KOOS-12 is a short self-reported measure that assesses patient's opinions about the difficulties they experience due to problems with their knee and also covers aspects of pain, functional limitations, and knee-related quality of life. • The Spanish version of KOOS-12 questionnaire is a valid instrument for measuring the patients' opinions about their knee and associated problems, and is both reliable and responsiveness to change.

The assessment of paranoia in young people: Item and test properties of the Bird Checklist of Adolescent Paranoia

BACKGROUND

Precise assessment tools for psychotic experiences in young people may help identify symptoms early and facilitate advances in treatment. In this study we provide an exemplar - with a paranoia scale for youth - for improving measurement precision for psychotic experiences using item response theory (IRT). We evaluate the psychometric properties of the new measure, test for measurement invariance, and assess its potential for computerised adaptive testing (CAT).

METHOD

The 18-item Bird Checklist of Adolescent Paranoia (B-CAP) was completed by 1102 adolescents including 301 patients with mental health problems and 801 from the general population. After excluding outliers (n = 10), IRT was used to examine item properties, test reliability, and measurement invariance. The properties of an adaptive B-CAP were assessed using a simulation of 10,000 responses.

RESULTS

All B-CAP items were highly discriminative (a = 1.14-2.77), whereby small shifts in paranoia led to a higher probability of item endorsement. Test reliability was high (a > 0.90) across a wide range of paranoia severity (θ = -0.45-3.36), with the greatest precision at elevated levels. All items were invariant for gender, age, and population groups. The simulated adaptive B-CAP performed with high accuracy and required only 5-6 items at higher levels of paranoia severity.

CONCLUSIONS

The B-CAP is a reliable assessment tool with excellent psychometric properties to assess both non-clinical and clinical levels of paranoia in young people, with potential as an efficient adaptive test. In future, these approaches could be used to develop a multidimensional CAT to assess the full range of psychotic experiences in youth.

Validity of the cold pressor test and pain sensitivity questionnaire via online self-administration

To determine the feasibility of complex home-based phenotyping, 1,876 research participants from the customer base of 23andMe completed an online version of a Pain Sensitivity Questionnaire (PSQ) as well as a cold pressor test (CPT) which is used in clinical assessments of pain. Overall our online version of the PSQ performed similarly to the original pen-and-paper version. Construct validity of the PSQ total was demonstrated by internal consistency and consistent discrimination between more and less painful items. Criterion validity was demonstrated by correlation with pain sensitivity as measured by the CPT. Within the same cohort we performed a cold pressor test using a layperson description and household equipment. Comparison with published reports from controlled studies revealed similar distributions of cold pain tolerance times (i.e., time elapsed before removing the hand from the water). Of those who elected to participate in the CPT, a large majority of participants did not report issues with the test procedure or noncompliance with the instructions (97%). We confirmed a large sex difference in CPT thresholds in line with published data, such that women removed their hands from the water at a median of 54.2 seconds, with men lasting for a median time of 82.7 seconds (Kruskal-Wallis statistic, p < 0.0001), but other factors like age or current pain treatment were at most weakly associated, and inconsistently between men and women. We introduce a new paradigm for performing pain testing, called testing@home, that, in the case of cold nociception, showed comparable results to studies conducted under controlled conditions and supervision of a health care professional.

High Levels of Glycosaminoglycans in the Urines of Children with Attention-Deficit/Hyperactivity Disorder (ADHD)

Attention-deficit/hyperactivity disorder (ADHD) is a common neurobehavioral/neurodevelopmental disorder. Some early studies indicated that increased intake of added sugars might have a role in ADHD. In the present study, we tested this possibility by evaluating the urinary excretion of oligosaccharides and glycosaminoglycans (GAGs) in ADHD and control subjects. Forty ADHD subjects matched with 34 controls were enrolled in the study. The subjects underwent a standardized dietary regimen. The urine levels of oligosaccharides and GAGs were quantified biochemically, and their covariance and association were evaluated statistically. Fructose (21/40, 52.5%), maltose (26/40, 65%), galactose (30/40, 75%), and lactose (38/40, 95%) excretions were frequently found in the urine of ADHD subjects (p < 0.05), an excretion which does not occur normally. Furthermore, these subjects showed a pathologic tGAG (glycosaminoglycan) excretion (40/40, 100%). The present study supports the thesis that carbohydrate metabolism differs in ADHD subjects compared with control subjects.

Robust maximum marginal likelihood (RMML) estimation for item response theory models

Self-report data are common in psychological and survey research. Unfortunately, many of these samples are plagued with careless responses, due to unmotivated participants. The purpose of this study was to propose and evaluate a robust estimation method to detect careless or unmotivated responders, while leveraging item response theory (IRT) person-fit statistics. First, we outlined a general framework for robust estimation specific for IRT models. Subsequently, we conducted a simulation study covering multiple conditions in order to evaluate the performance of the proposed method. Ultimately, we showed that robust maximum marginal likelihood (RMML) estimation significantly improves detection rates for careless responders and reduces bias in item parameters across conditions. Furthermore, we applied our method to a real data set, to illustrate the utility of the proposed method. Our findings suggest that robust estimation coupled with person-fit statistics offers a powerful procedure to identify careless respondents for further review and to provide more accurate item parameter estimates in the presence of careless responses.

A content validity evaluation of the PEDI-CAT Speedy Mobility domain

Objective: To evaluate the content validity of the PEDI-CAT Speedy Mobility domain through analysis of item and content area exposure, score range and scoring precision.Methods: Retrospective analysis of 3,364 items from assessments (n = 301) completed from 2013 to 2017. Content validity was appraised through analysis of item and content area exposure (item, content area, response frequency), score range (floor and ceiling effect) and scoring precision (person fit, score reliability, item information function).Results: Sixty-five of the 75 general mobility items from the PEDI-CAT Mobility domain item bank were exposed. "Stands up from the middle of the floor" (68%) was the most frequently exposed non-mandatory item. Almost half (49%) of all items were from the Basic Mobility and Transfers content area. Scaled scores ranged from 26.77 to 69.40 with a floor (scores ≤27; n = 51, 17%) but no ceiling effect. Person fit statistics were acceptable for 238 (79%), suggesting limited outliers. Score reliability was sufficient with 68% of scores above threshold (>0.9). Item information function plot indicated less discriminating items at the lower end of the score range.Conclusion: Content is adequately and reliably measuring the intended construct, but additional items at the lower end of the scale could improve score precision.

Cronbach’s Alpha under Insufficient Effort Responding: An Analytic Approach

Surveys commonly suffer from insufficient effort responding (IER). If not accounted for, IER can cause biases and lead to false conclusions. In particular, Cronbach’s alpha has been empirically observed to either deflate or inflate due to IER. This paper will elucidate how IER impacts Cronbach’s alpha in a variety of situations. Previous results concerning internal consistency under mixture models are extended to obtain a characterization of Cronbach’s alpha in terms of item validities, average variances, and average covariances. The characterization is then applied to contaminating distributions representing various types of IER. The discussion will provide commentary on previous simulation-based investigations, confirming some previous hypotheses for the common types of IER, but also revealing possibilities from newly considered responding patterns. Specifically, it is possible that the bias can change from negative to positive (and vice versa) as the proportion of contamination increases.

A Practical Illustration of Methods to Deal with Potential Outliers: A Multiverse Outlier Analysis of Study 3 from Brummelman, Thomaes, Orobio de Castro, Overbeek, and Bushman ()

Recently, Brummelman, Thomaes, Orobio de Castro, Overbeek, and Bushman (2014: Study 3) demonstrated that inflated praise benefits challenge seeking of children with high self-esteem, but harms challenge seeking of children with low self-esteem. In the present paper, we examined the original data set on model-fit and prediction outliers according to various reasonable criteria and norms. Subsequently, we carried out a multiverse outlier re-analysis on the data of Brummelman and colleagues’ Study 3, employing the same analytical approach as the original authors did but excluding outliers. Out of the twelve re-analyses in the multiverse, six demonstrated that removing only a small number of outliers rendered the originally reported crucial interaction effect between self-esteem and type of praise non-significant and produced a sizeable reduction of the effect size. The present paper illustrates the use of reporting outlier analyses, which lies in allowing a critical evaluation of the empirical evidence and offering a more complete picture that enhances future studies in the field.