The Dunning-Kruger effect is (mostly) a statistical artefact: Valid approaches to testing the hypothesis with individual differences data

Prevalence of Dunning-Kruger effect in first semester medical students: a correlational study of self-assessment and actual academic performance

INTRODUCTION

The ability to self-assess is a crucial skill in identifying one's own strengths and weaknesses and in coordinating self-directed learning. The Dunning-Kruger effect occurs when limited knowledge causes individuals to overestimate their competence and underestimate others', leading to poor self-assessment and unrecognized incompetence. To serve as a foundation for developing strategies to improve self-assessment, the self-assessment abilities of first-semester students were assessed.

METHODS

In the final weeks of the summer 2021, winter 2021/22, and summer 2022 semesters, the academic performance (oral anatomy exam) of first semester students was assessed (0-15 points). Before the exam results were announced, students were asked to self-assess their performance.

RESULTS

Exam scores (M = 10.64, SD = 2.95) and self-assessed scores (M = 10.38, SD = 2.54) were comparable. The absolute difference between them, as a measure of self-assessment ability ranged from - 9 to + 9 points (M = -0.26, SD = 2.59). Among participants (N = 426), 18.5% assessed themselves accurately, 35.5% overestimated, and 46.0% underestimated their performance. The correlation between actual score and self-assessment was ρ = -0.590 (p < 0.001), reflecting the Dunning-Kruger effect. When separated by gender, correlation for females was ρ = -0.591 (p < 0.001), and for males ρ = -0.580 (p < 0.001).

CONCLUSIONS

Realistic self-assessment is a challenge for first-semester students. The data indicate that females tend to overestimate their performance while males underestimate theirs. A pronounced Dunning-Kruger effect is evident in both genders, with significant negative correlations between self-assessment and actual performance. There are several reasons for the occurrence of the Dunning-Kruger effect. Considering that the COVID-19 pandemic influenced learning environments, collaborative learning was significantly restricted. The lack of opportunities for comparison could potentially lead to unrealistic self-assessment.

No strong support for a Dunning-Kruger effect in creativity: analyses of self-assessment in absolute and relative terms

Competencies related to the evaluation of own cognitive processes, called metacognitive monitoring, are crucial as they help decide whether to persist in or desist from cognitive efforts. One of the most well-known phenomena in this context-the Dunning-Kruger effect-is that less skilled people tend to overestimate their performance. This effect has been reported for various kinds of performance including creativity. More recently, however, it has been suggested that this phenomenon could be a statistical artifact caused by the better-than-average effect and by regression toward the mean. Therefore, we examined the Dunning-Kruger effect in the context of creative thinking performance (i.e., divergent thinking ability) across two studies (Study 1: N = 425; Study 2: N = 317) and applied the classical quartile-based analysis as well as newly recommended, advanced statistical approaches: the Glejser test of heteroscedasticity and nonlinear quadratic regression. We found that the results indeed depended on the employed statistical method: While classical analyses supported the Dunning-Kruger effect across all conditions, it was not consistently supported by the more advanced statistical methods. These findings are in line with recent work challenging certain assumptions of the Dunning-Kruger effect and we discuss factors that undermine accurate self-assessments, especially in the context of creative performance.

Overlapping yet dissociable contributions of superiority illusion features to Ponzo illusion strength and metacognitive performance

Humans are typically inept at evaluating their abilities and predispositions. People dismiss such a lack of metacognitive insight into their capacities while even enhancing (albeit illusorily) self-evaluation such that they should have more desirable traits than an average peer. This superiority illusion helps maintain a healthy mental state. However, the scope and range of its influence on broader human behavior, especially perceptual tasks, remain elusive. As belief shapes the way people perceive and recognize, the illusory self-superiority belief potentially regulates our perceptual and metacognitive performance. In this study, we used hierarchical Bayesian estimation and machine learning of signal detection theoretic measures to understand how the superiority illusion influences visual perception and metacognition for the Ponzo illusion. Our results demonstrated that the superiority illusion correlated with the Ponzo illusion magnitude and metacognitive performance. Next, we combined principal component analysis and cross-validated regularized regression (relaxed elastic net) to identify which superiority components contributed to the correlations. We revealed that the "extraversion" superiority dimension tapped into the Ponzo illusion magnitude and metacognitive ability. In contrast, the "honesty-humility" and "neuroticism" dimensions only predicted Ponzo illusion magnitude and metacognitive ability, respectively. These results suggest common and distinct influences of superiority features on perceptual sensitivity and metacognition. Our findings contribute to the accumulating body of evidence indicating that the leverage of superiority illusion is far-reaching, even to visual perception.

Activity space during treatment with medication for opioid use disorder: Relationships with personality, mood, and drug use

INTRODUCTION

Activity space in people with substance use disorders (SUDs) has been assessed for theoretical reasons and for detection/prevention of relapse. In this observational study, we relate passively obtained activity space measures to mental states and behaviors relevant to the success of treatment for opioid use disorder. Our long-term goal is to use such data to assess risk in real time and to recognize when SUD patients might benefit from a just-in-time intervention.

METHODS

We used GPS data from 238 urban residents in the first 16 weeks of stabilization on medication for opioid use disorder to test preregistered hypotheses about activity space (distance traveled, number of locations, time spent moving, and psychosocial-hazard levels of neighborhoods where participants spent time) in relation to certain static variables (personality, mood propensities) and time-varying treatment-relevant behaviors such as craving and use of opioids and cocaine.

RESULTS

The most consistent findings were that 1) mobility decreased over the course of the study; 2) neuroticism was associated with overall lower mobility; 3) trait-like positive mood (averaged from momentary ratings) was associated with higher mobility; 4) participants who used cocaine more frequently had lower mobility; 5) early in treatment, participants spent less time moving (i.e., were more sedentary) on days when they were craving. Some of these findings were in the expected direction (i.e., the ones involving neuroticism and positive mood), and some were opposite to the expected direction (i.e., we expected cocaine use to be associated with higher mobility); others (e.g., changes in mobility over time or in relation to craving) involved nondirectional hypotheses.

CONCLUSIONS

Real-time information that patients actively provide is valuable for assessing their current state, but providing this information can be burdensome. The current results indicate that certain static or passively obtained data (personality variables and GPS-derived mobility information) are relevant to time-varying, treatment-relevant mental states and drug-related behavior, and therefore might be useful when incorporated into algorithms for detecting need for intervention in real time. Further research should assess how population-specific these relationships are, and how these passive measures can best be combined with low temporal-density, actively-provided data to obtain valid, reliable assessments with minimal burden.

Unskilled, underperforming, or unaware? Testing three accounts of individual differences in metacognitive monitoring

Many studies show that competence (e.g., skill, expertise, natural ability) influences individuals' capabilities of monitoring their item-level performance. However, debate persists about how best to explain these individual differences in metacognition. The competence-based account ascribes differences in monitoring to individuals' objective ability level, arguing that the same skills necessary to perform a task are required to effectively monitor performance. The performance-based account attributes differences in monitoring to changes in overall task performance - no individual differences in competence required. Finally, the metacognitive awareness account proposes that alignment between an individuals' self-assessed and objective ability leads to differences in monitoring. In this study, 603 participants completed a self-assessment of face recognition ability, a lineup identification task, and an objective assessment of face recognition ability. We manipulated the number of encoding repetitions and delay between encoding and test to produce varying levels of task performance across objective face recognition ability. Following each lineup decision, participants provided both a numeric confidence rating and a written expression of verbal confidence. We transformed verbal confidence into a quantitative value using machine learning techniques. When matched on overall identification accuracy, objectively stronger face recognizers used numeric and verbal confidence that a) better discriminates between correct and filler lineup identifications than weaker recognizers, and b) shows better calibration to accuracy. Participants with greater self-assessed ability used higher levels of confidence, irrespective of trial accuracy. These results support the competence-based account.

A new variation of modern prejudice: young Korean men's anti-feminism and male-victim ideology

In South Korea, anti-feminism is now rapidly spreading online among young men, who have started to identify themselves as a social minority or "victims" of female power. Despite its ramifications, theoretically, anti-feminism is indistinct from the racism and sexism of White men that emerged more than half a century ago. In view of this, it shares the same root as typical modern racism or sexism, although it appears to be a novel phenomenon. Such a hypothesis was buttressed by quantifying the attitudes of anti-feminists toward various outgroups based on the transference of prejudice theory. Moreover, the subtle sexist undertones hidden in their arguments have been discussed using various psychological theories and empirical data/statistics. Additionally, various potential factors that may shape or accelerate their attitudes or behaviors have been discussed on the basis of the threat-defense theory. Through comprehensive literature review based on this theory, this study proposes the features related to Korean anti-feminism, encompassing behavioral/situational (overindulging violent or degrading Internet contents, verbal aggression), relational/epistemic (ostracism, attachment insecurity, pseudo-rationalism), and group-level (provocative interactions, polarization) attributes, some of which may also influence groups other than young men and ingrain or exacerbate the extreme ideologies of other groups, including young women. Scrutinizing Korean online anti-feminism and male-victim ideology may improve our understanding of the psychological origins of various social extremities or radical ideologies beyond cultural barriers.

Intermediate levels of scientific knowledge are associated with overconfidence and negative attitudes towards science

Overconfidence is a prevalent problem and it is particularly consequential in its relation with scientific knowledge: being unaware of one's own ignorance can affect behaviours and threaten public policies and health. However, it is not clear how confidence varies with knowledge. Here, we examine four large surveys, spanning 30 years in Europe and the United States and propose a new confidence metric. This metric does not rely on self-reporting or peer comparison, operationalizing (over)confidence as the tendency to give incorrect answers rather than 'don't know' responses to questions on scientific facts. We find a nonlinear relationship between knowledge and confidence, with overconfidence (the confidence gap) peaking at intermediate levels of actual scientific knowledge. These high-confidence/intermediate-knowledge groups also display the least positive attitudes towards science. These results differ from current models and, by identifying specific audiences, can help inform science communication strategies.

Blocked Presentation Leads Participants to Overutilize Domain Familiarity as a Cue for Judgments of Learning (JOLs)

The accuracy of judgments of learning (JOLs) is vital for efficient self-regulated learning. We examined a situation in which participants overutilize their prior knowledge of a topic ("domain familiarity") as a basis for JOLs, resulting in substantial overconfidence in topics they know the most about. College students rank ordered their knowledge across ten different domains and studied, judged, and then completed a test on facts from those domains. Recall and JOLs were linearly related to self-rated knowledge, as was overconfidence: participants were most overconfident for topics they knew more about, indicating the overutilization of domain familiarity as a cue for JOLs. We examined aspects of the task that might contribute to this pattern, including the order of the task phases and whether participants studied the facts blocked by topic. Although participants used domain familiarity as a cue for JOLs regardless of task design, we found that studying facts from multiple topics blocked by topic led them to overutilize this cue. In contrast, whether participants completed the rank ordering before studying the facts or received a warning about this tendency did not alter the pattern. The relative accuracy of participants' JOLs, however, was not related to domain familiarity under any conditions.

AI model GPT-3 (dis)informs us better than humans

Artificial intelligence (AI) is changing the way we create and evaluate information, and this is happening during an infodemic, which has been having marked effects on global health. Here, we evaluate whether recruited individuals can distinguish disinformation from accurate information, structured in the form of tweets, and determine whether a tweet is organic or synthetic, i.e., whether it has been written by a Twitter user or by the AI model GPT-3. The results of our preregistered study, including 697 participants, show that GPT-3 is a double-edge sword: In comparison with humans, it can produce accurate information that is easier to understand, but it can also produce more compelling disinformation. We also show that humans cannot distinguish between tweets generated by GPT-3 and written by real Twitter users. Starting from our results, we reflect on the dangers of AI for disinformation and on how information campaigns can be improved to benefit global health.

Face matching and metacognition: investigating individual differences and a training intervention

Background

Although researchers have begun to consider metacognitive insight during face matching, little is known about the underlying mechanism. Here, I investigated whether objective ability, as well as self-assessed ability, were able to predict metacognitive performance, that is, the ability to differentiate correct and incorrect responses in terms of confidence. In addition, I considered whether a training intervention resulted in improvements to both face matching performance and metacognitive insight.

Methods

In this experiment (N = 220), participants completed a face matching task, with either a diagnostic feature training course or a control course presented at the halfway point. In addition, a second face matching task, as well as a self-report questionnaire regarding ability, were completed to provide measures of objective and self-assessed ability respectively.

Results

Higher self-assessed ability with faces, as well as higher objective ability with face matching, predicted better metacognitive performance, i.e., greater confidence in correct, in comparison with incorrect, responses. This pattern of results was evident both when objective ability was measured through performance on the same task used to measure metacognitive insight and when a different task was used. Finally, the training intervention failed to produce improvements in face matching performance and showed no evidence of altering metacognitive ability.

Discussion

The current work begins to address the mechanism underlying individual differences in metacognitive insight during face matching. Although support was provided for a competence-based account, where better face matchers showed greater performance on the task and were more successful in monitoring their performance, further work might focus on decoupling task performance and competence in order to more conclusively explain why some people are more insightful than others.

The role of personality, authoritarianism and cognition in the United Kingdom's 2016 referendum on European Union membership

Introduction

The results of the United Kingdom's 2016 referendum on European Union (EU) membership have highlighted deep societal divides. In six studies, we examined the role of personality traits, cognition and cognitive biases in relation to referendum voters' choices.

Methods

A total of 11,225 participants completed questionnaires and controlled experiments, which assessed differences in personality traits, levels of authoritarianism, numeracy, thinking styles, and susceptibility to cognitive biases including ideologically motivated numeracy and reasoning, framing, and the Dunning-Kruger effect.

Results

Participants expressing an intent to vote to leave the EU reported significantly higher levels of authoritarianism and conscientiousness, and lower levels of openness and neuroticism than voters expressing an intent to vote to remain in the EU. When compared with Remain voters, Leave voters displayed significantly lower levels of numeracy and appeared more reliant on impulsive System 1 thinking. In the experimental studies, voters on both sides were found to be susceptible to the cognitive biases tested, with a general trend for Leave voters to show more bias than Remain voters.

Discussion

These results raise important questions regarding the use and framing of numerical and non-numerical data for public consumption.

Pitfalls in Post Hoc Analyses of Population Receptive Field Data

Data binning involves grouping observations into bins and calculating bin-wise summary statistics. It can cope with overplotting and noise, making it a versatile tool for comparing many observations. However, data binning goes awry if the same observations are used for binning (selection) and contrasting (selective analysis). This creates circularity, biasing noise components and resulting in artifactual changes in the form of regression towards the mean. Importantly, these artifactual changes are a statistical necessity. Here, we use (null) simulations and empirical repeat data to expose this flaw in the scope of post hoc analyses of population receptive field data. In doing so, we reveal that the type of data analysis, data properties, and circular data cleaning are factors shaping the appearance of such artifactual changes. We furthermore highlight that circular data cleaning and circular sorting of change scores are selection practices that result in artifactual changes even without circular data binning. These pitfalls might have led to erroneous claims about changes in population receptive fields in previous work and can be mitigated by using independent data for selection purposes. Our evaluations highlight the urgency for us researchers to make the validation of analysis pipelines standard practice.

A Statistical Explanation of the Dunning-Kruger Effect

An explanation of the Dunning-Kruger effect is provided which does not require any psychological explanation, because it is derived as a statistical artifact. This is achieved by specifying a simple statistical model which explicitly takes the (random) boundary constraints into account. The model fits the data almost perfectly.

JEL Classification

A22; C24; C91; D84; D91; I21.

Less-Intelligent and Unaware? Accuracy and Dunning–Kruger Effects for Self-Estimates of Different Aspects of Intelligence

People’s perceptions of their intelligence correlate only moderately with objective intelligence measures. On average, people overestimate themselves. According to the popular Dunning–Kruger effect, this is particularly true for low performers: across many domains, those in the lowest quartile overestimate their abilities the most. However, recent work using improved statistical approaches found little support for a Dunning–Kruger effect in general intelligence. We investigated accuracy and Dunning–Kruger effects for self-estimates of general, verbal, numerical, and spatial intelligence—domains that differed in how well they can be judged in the past. A total of 281 participants completed self-estimates and intelligence measures online. Self-estimates showed mostly moderate correlational accuracy that was slightly higher for numerical intelligence and lower for verbal intelligence. Across domains, participants rated their intelligence as above average. However, as their intelligence was indeed high, this was not an overestimation. While standard analyses indicated Dunning–Kruger effects in general, verbal, and spatial intelligence, improved statistical methods only yielded some support for one in verbal intelligence: people with lower verbal intelligence tended to have less self-knowledge about it. The generalizability of these findings is limited to young, highly educated populations. Nevertheless, our results contribute to a growing literature questioning the generality of the Dunning–Kruger effect.

Metacognition during unfamiliar face matching

Kruger and Dunning (1999) described a metacognitive bias in which insight into performance is linked to competence: poorer performers are less aware of their mistakes than better performers. Competence-based insight has been argued to apply generally across task domains, including a recent report investigating social cognition using a variety of face-matching tasks. Problematically, serious statistical and methodological criticisms have been directed against the traditional method of analysis used by researchers in this field. Here, we further illustrate these issues and investigate new sources of insight within unfamiliar face matching. Over two experiments (total N = 1077), where Experiment 2 was a preregistered replication of the key findings from Experiment 1, we found that insight into performance was multi-faceted. Participants demonstrated insight which was not based on competence, in the form of accurate updating of estimated performance. We also found evidence of insight which was based on competence: the difference in confidence on correct versus incorrect trials increased with competence. By providing ways that we can move beyond problematic, traditional approaches, we have begun to reveal a more realistic story regarding the nature of insight into face perception.

Enhancing student scientific literacy through participation in citizen science focused on companion animal behavior

The Covid-19 pandemic served as the impetus to implement activities designed to engage students in the remote instructional environment while simultaneously developing scientific literacy skills. In a high enrollment general education animal science course, numerous activities were designed to improve scientific literacy. These included specifically developed videos covering strategies for reading published science literature, the utilization of topically relevant scientific articles that captured student interest, and engaging students in a citizen science exercise on whether dogs align themselves to the Earth's magnetic field during excretion behavior. Employing pre- and post-self-perception surveys coupled with tasking students to apply their scientific literacy skills in an assessment scenario demonstrated that students' self-perception of their scientific literacy improved 30% (P < 0.05) with approximately 80% of students accurately applying their literacy skills. The citizen science study on excretory behavior was modeled on previously published findings thereby providing an opportunity to validate the published work which had indicated that dogs align their bodies in a North–South axis during excretion. The present study did not demonstrate preferential alignment to any geomagnetic orientation which emphasized to the students the need for scientific replication. Inclusion of simple activities that were relevant to students' daily lives, and providing interpretive context for those activities, resulted in improved self-perceived scientific literacy.

Mathematics Clusters Reveal Strengths and Weaknesses in Adolescents' Mathematical Competencies, Spatial Abilities, and Mathematics Attitudes

Pre-algebra mathematical competencies were assessed for a large and diverse sample of sixth graders (n = 1,926), including whole number and fractions arithmetic, conceptual understanding of equality and fractions magnitudes, and the fractions number line. The goal was to determine if there were clusters of students with similar patterns of pre-algebra strengths and weaknesses and if variation between clusters was related to mathematics attitudes, anxiety, or for a subsample (n = 342) some combination of intelligence, working memory, or spatial abilities. Critically, strengths and weaknesses were not uniform across the three identified clusters. Lower-performing students had pronounced deficits in their understanding of mathematical equality, fractions magnitudes, and the fractions number line. Higher-performing students had particular advantages in whole number and fractions arithmetic, and the fractions number line. Students could be reliably placed into clusters based on their mathematics self-efficacy and a combination of intelligence and spatial abilities. The results contribute to our understanding of key aspects of students' mathematical development, highlight areas in need of intervention for at-risk students, and identify cognitive areas in which scaffolds might be incorporated into these interventions.

Overconfidence in news judgments is associated with false news susceptibility

We examine the role of overconfidence in news judgment using two large nationally representative survey samples. First, we show that three in four Americans overestimate their relative ability to distinguish between legitimate and false news headlines; respondents place themselves 22 percentiles higher than warranted on average. This overconfidence is, in turn, correlated with consequential differences in real-world beliefs and behavior. We show that overconfident individuals are more likely to visit untrustworthy websites in behavioral data; to fail to successfully distinguish between true and false claims about current events in survey questions; and to report greater willingness to like or share false content on social media, especially when it is politically congenial. In all, these results paint a worrying picture: The individuals who are least equipped to identify false news content are also the least aware of their own limitations and, therefore, more susceptible to believing it and spreading it further.

Neural correlates of the Dunning-Kruger effect

The Dunning-Kruger effect (DKE) is a metacognitive phenomenon of illusory superiority in which individuals who perform poorly on a task believe they performed better than others, yet individuals who performed very well believe they under-performed compared to others. This phenomenon has yet to be directly explored in episodic memory, nor explored for physiological correlates or reaction times. We designed a novel method to elicit the DKE via a test of item recognition while electroencephalography (EEG) was recorded. Throughout the task, participants were asked to estimate the percentile in which they performed compared to others. Results revealed participants in the bottom 25th percentile over-estimated their percentile, while participants in the top 75th percentile under-estimated their percentile, exhibiting the classic DKE. Reaction time measures revealed a condition-by-group interaction whereby over-estimators responded faster than under-estimators when estimating being in the top percentile and responded slower when estimating being in the bottom percentile. Between-group EEG differences were evident between over-estimators and under-estimators during Dunning-Kruger responses, which revealed FN400-like effects of familiarity supporting differences for over-estimators, whereas "old-new" memory event-related potential effects revealed a late parietal component associated with recollection-based processing for under-estimators that was not evident for over-estimators. Findings suggest over- and under-estimators use differing cognitive processes when assessing their performance, such that under-estimators may rely on recollection during memory while over-estimators may draw upon excess familiarity when over-estimating their performance. Episodic memory thus appears to play a contributory role in metacognitive judgements of illusory superiority.