Measuring cognitive reflection without maths: Development and validation of the verbal cognitive reflection test

Fast & slow decisions under risk: Intuition rather than deliberation drives advantageous choices

Would you take a gamble with a 10% chance to gain $100 and a 90% chance to lose $10? Even though this gamble has a positive expected value, most people would avoid taking it given the high chance of losing money. Popular "fast-and-slow" dual process theories of risky decision making assume that to take expected value into account and avoid a loss aversion bias, people need to deliberate. In this paper we directly test whether reasoners can also consider expected value benefit intuitively, in the absence of deliberation. To do so, we presented participants with bets and lotteries in which they could choose between a risky expected-value-based choice and a safe loss averse option. We used a two-response paradigm where participants made two choices in every trial: an initial intuitive choice under time-pressure and cognitive load and a final choice without constraints where they could freely deliberate. Results showed that in most trials participants were loss averse, both in the intuitive and deliberate stages. However, when people opted for the expected-value-based choice after deliberating, they had predominantly already arrived at this choice intuitively. Additionally, loss averse participants often showed an intuitive sensitivity to expected value (as reflected in decreased confidence). Overall, these results suggest that deliberation is not the primary route for expected-value-based responding in risky decision making. Risky decisions may be better conceptualized as an interplay between different types of "fast" intuitions rather than between two different types of "fast" and "slow" thinking per se.

Applying process dissociation to self-sacrificial moral dilemmas: Extending the dual-process model

There is considerable evidence linking cognitive reflection with utilitarian judgments in dilemmas that involve sacrificing someone else for the greater good. However, the evidence is mixed on the question of whether cognitive reflection is associated with utilitarian judgments in self-sacrificial dilemmas. We employed process dissociation to extract a self-sacrificial utilitarian (SU) parameter, an altruism (A) parameter, an other-sacrificial (OU) utilitarian parameter, and a deontology (D) parameter. In Study 1, the cognitive reflection test (CRT) positively correlated with both SU and OU (replicated in Studies 2 and 4, pre-registered). In Study 2, we found that instructing participants to rely on reason increased SU and OU (replicated in Study 4, pre-registered). In Study 3, we found that SU and OU positively correlated with giving in the single-game version of the public goods game (replicated in Study 4, pre-registered), which provides behavioral validation that they are genuine moral tendencies. Together, these studies constitute strong cumulative evidence that SU and OU are both valid measures that are associated with reliance on cognitive reflection.

Sex Differences in Cognitive Reflection: A Meta-Analysis

The current study presents a meta-analytic review of the differences between men and women in cognitive reflection (CR). The study also explores whether the type of CR test (i.e., numerical tests and verbal tests) moderates the relationship between CR and sex. The results showed that men score higher than women on CR, although the magnitude of these differences was small. We also found out that the type of CR test moderates the sex differences in CR, especially in the numerical tests. In addition, the results showed that the length of numerical tests (i.e., number of items) does not affect the differences between men and women in CR. Finally, the implications of these results are discussed, and future research is suggested.

Improving patient understanding of prenatal screening tests: Using naturally sampled frequencies, pictures, and accounting for individual differences

Objective

Health professionals must communicate prenatal screening test results effectively to patients, but these tests involve very low prevalence and high false positive risks; a situation known to be difficult for people to understand.

Methods

The present experiments evaluated the effectiveness of presenting prenatal screening test results for Trisomy 21, Trisomy 13, or DiGeorge Syndrome, using either naturally sampled frequencies or standard percentages. Participants were given a task of interpreting the posterior probability of an embryo having the condition.

Results

People were significantly better with naturally sampled frequencies. Numerical literacy and visuospatial ability significantly accounted for individual differences in performance within conditions. Participants nevertheless did not differ in ratings of how useful the different presentation formats were, suggesting a lack of awareness of how format influenced understanding. These results held regardless of whether the respondents were undergraduates (Experiment 1) or members of the general population recruited online (Experiment 2).

Conclusion

Using naturally sampled frequencies improves patient understanding of prenatal screening tests results, with low cost of implementation.

Innovation

Using realistic prenatal screening test results, these results show how to improve patient counseling via the use of naturally sampled frequencies.

Thinking false and slow: Implausible beliefs and the Cognitive Reflection Test

Why do people believe implausible claims like conspiracy theories, pseudoscience, and fake news? Past studies using the Cognitive Reflection Test (CRT) suggest that implausible beliefs may result from an unwillingness to effortfully process information (i.e., cognitive miserliness). Our analysis (N = 664) tests this account by comparing CRT performance (total score, number and proportion of incorrect intuitive responses, and completion time) for endorsers and non-endorsers of implausible claims. Our results show that endorsers performed worse than non-endorsers on the CRT, but they took significantly longer to answer the questions and did not make proportionally more intuitive mistakes. Endorsers therefore appear to process information effortfully but nonetheless score lower on the CRT. Poorer overall CRT performance may not necessarily indicate that those who endorse implausible beliefs have a more reflexive, intuitive, or non-analytical cognitive style than non-endorsers.

Why Should We Study the Foreign Language Effect: Debiasing through Affecting Metacognition?

Debiasing is a method of improving people's decisions by reducing their reliance on salient intuitions causing them to behave suboptimally or biasedly. However, many of the known debiasing techniques have limited effectiveness or can only remedy a one-shot decision, rather than having a lasting impact. In this work, I focus on the role of metacognition in debiasing decision-making and discuss how it can be better understood through the lens of the foreign language effect. The foreign language effect suggests that using a foreign language can sometimes benefit people's decision-making without providing them with additional information or instructions regarding the task. However, we do not fully understand how the foreign language effect works and its limitations. I conclude by urging scientists to research this effect with the hope of having a lasting positive impact on society.

Numeracy, gist, literal thinking and the value of nothing in decision making

The onus on the average person is greater than ever before to make sense of large amounts of readily accessible quantitative information, but the ability and confidence to do so are frequently lacking. Many people lack practical mathematical skills that are essential for evaluating risks, probabilities and numerical outcomes such as survival rates for medical treatments, income from retirement savings plans or monetary damages in civil trials. In this Review, we integrate research on objective and subjective numeracy, focusing on cognitive and metacognitive factors that distort human perceptions and foment systematic biases in judgement and decision making. Paradoxically, an important implication of this research is that a literal focus on objective numbers and mechanical number crunching is misguided. Numbers can be a matter of life and death but a person who uses rote strategies (verbatim representations) cannot take advantage of the information contained in the numbers because 'rote' strategies are, by definition, processing without meaning. Verbatim representations (verbatim is only surface form, not meaning) treat numbers as data as opposed to information. We highlight a contrasting approach of gist extraction: organizing numbers meaningfully, interpreting them qualitatively and making meaningful inferences about them. Efforts to improve numerical cognition and its practical applications can benefit from emphasizing the qualitative meaning of numbers in context - the gist - building on the strengths of humans as intuitive mathematicians. Thus, we conclude by reviewing evidence that gist training facilitates transfer to new contexts and, because it is more durable, longer-lasting improvements in decision making.

Overconfidence in the Cognitive Reflection Test: Comparing Confidence Resolution for Reasoning vs. General Knowledge

This research examines the metacognitive awareness that people have about their reasoning performance in the Cognitive Reflection Test (CRT). The first two studies compare confidence judgments about the CRT vs. general knowledge (GK) questions. Results show that (1) people are generally able to discriminate between correct and incorrect answers, but this ability is far from perfect, and it is greater for GK questions than for CRT problems. Indeed, and strikingly, (2) incorrect responses to CRT problems are produced with approximately the same level of confidence as correct responses to GK questions. However, (3) even though confidence is high for incorrect responses to CRT problems, it is even higher for correct responses. The results of two additional studies show that these differences in confidence are ultimately related to the conflict that CRT problems pose between intuition and deliberation. These findings have implications for the possibility of implicit error monitoring and dual-process models of overconfidence.

Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test

The standard interpretation of cognitive reflection tests assumes that correct responses are reflective and lured responses are unreflective. However, prior process-tracing of mathematical reflection tests has cast doubt on this interpretation. In two studies (N = 201), we deployed a validated think-aloud protocol in-person and online to test how this assumption is satisfied by the new, validated, less familiar, and non-mathematical verbal Cognitive Reflection Test (vCRT). Verbalized thoughts in both studies revealed that most (but not all) correct responses involved reflection and that most (but not all) lured responses lacked reflection. The think-aloud protocols seemed to reflect business-as-usual performance: thinking aloud did not disrupt test performance compared to a control group. These data suggest that the vCRT usually satisfies the standard interpretation of the reflection tests (albeit not without exceptions) and that the vCRT can be a good measure of the construct theorized by the two-factor explication of 'reflection' (as deliberate and conscious).

Scientific reasoning is associated with rejection of unfounded health beliefs and adherence to evidence-based regulations during the Covid-19 pandemic

Scientific reasoning and trust in science are two facets of science understanding. This paper examines the contribution of science understanding, over and above analytic thinking, to the endorsement of conspiracy and pseudoscientific beliefs about COVID-19 and behavioral intentions to engage in the recommended preventive behavior. We examined the direct and indirect effects of science understanding on normative health behavior in a representative sample of the Slovak population (N = 1024). The results showed more support for the indirect pathway: individuals with a better understanding of science generally had fewer epistemically suspect beliefs and as a consequence tended to behave more in line with the evidence-based guidelines and get vaccinated. Neither scientific reasoning nor trust in science directly predicted non-compliance with preventive measures, but analytic thinking correlated positively with non-compliance with preventive measures. The strongest predictor of epistemically suspect beliefs was trust in science, which also directly predicted the intention to get vaccinated against COVID-19. Therefore, reasoning about which experts or sources to believe (second-order scientific reasoning) has become more important than directly evaluating the original evidence (first-order scientific reasoning).

Supplementary information

The online version contains supplementary material available at 10.1007/s12144-023-04284-y.

Computational underpinnings of partisan information processing biases and associations with depth of cognitive reasoning

Despite unprecedented access to information, partisans increasingly disagree about basic facts that are backed by data, posing a serious threat to a democracy that relies on finding common ground based on objective truths. We examine the underpinnings of this phenomenon using drift diffusion modeling (DDM). Partisans (N = 148) completed a sequential sampling task where they evaluated the honesty of Democrat or Republican politicians during a debate based on fact-check scores. We found that partisans required less and weaker evidence to correctly categorize the ingroup as more honest, and were more accurate on trials when the ingroup candidate was more honest, compared to the outgroup. DDM revealed that such tendencies arise from both a prior preference for categorizing the ingroup as more honest (i.e., biased starting point) and more precise accumulation of information favoring the ingroup candidate compared to the outgroup (i.e., biased drift rate). Moreover, individual differences in cognitive reasoning moderated task performance for the most devoted partisans and maintained divergent associations with the DDM parameters. This suggests that partisans may reach biased conclusions via different pathways depending on their depth of cognitive reasoning. These findings provide key insights into the mechanisms driving partisan divides in polarized environments, and can inform interventions that reduce impasse and conflict.

Cognitive reflection test: The effects of the items sequence on scores and response time

This paper aims to expand the literature on the determinants of the Cognitive Reflection Test scores, exploring the effects that the items sequence has on (1) Cognitive Reflection Test scores, (2) response time, (3) the relationship between Cognitive Reflection Test scores and response time, and (4) Cognitive Reflection scores, response time, and the relationship between both variables on men and women. The current study also explored the sex differences on Cognitive Reflection Test and response time according to items sequence. The results showed that manipulating the items sequence, the performance on the Cognitive Reflection Test improved significantly, but the response time were not significantly affected, although the results suggest that first items of the sequence could be working as training items. A positive relationship between Cognitive Reflection Test scores and response time was also found, except when the scores were maximized. Finally, some differences between men and women on the results were also found. The implications of these findings are discussed.

Superlatives, clickbaits, appeals to authority, poor grammar, or boldface: Is editorial style related to the credibility of online health messages?

Adolescents, as active online searchers, have easy access to health information. Much health information they encounter online is of poor quality and even contains potentially harmful health information. The ability to identify the quality of health messages disseminated via online technologies is needed in terms of health attitudes and behaviors. This study aims to understand how different ways of editing health-related messages affect their credibility among adolescents and what impact this may have on the content or format of health information. The sample consisted of 300 secondary school students (M_age = 17.26; SD_age = 1.04; 66.3% female). To examine the effects of manipulating editorial elements, we used seven short messages about the health-promoting effects of different fruits and vegetables. Participants were then asked to rate the message’s trustworthiness with a single question. We calculated second-order variable sensitivity as the derivative of the trustworthiness of a fake message from the trustworthiness of a true neutral message. We also controlled for participants’ scientific reasoning, cognitive reflection, and media literacy. Adolescents were able to distinguish overtly fake health messages from true health messages. True messages with and without editorial elements were perceived as equally trustworthy, except for news with clickbait headlines, which were less trustworthy than other true messages. The results were also the same when scientific reasoning, analytical reasoning, and media literacy were considered. Adolescents should be well trained to recognize online health messages with editorial elements characteristic of low-quality content. They should also be trained on how to evaluate these messages.

I Think I Should Get Vaccinated, I Feel I Should Not. Individual Differences in Information Processing and Vaccination Behavior (COVID-19)

Following the outbreak of the COVID-19 pandemic, the scientific community responded promptly by developing effective vaccines. Still, even though effective vaccines against COVID-19 became available, many people did not seem to be in a rush to become immunized. Community protection can be enhanced if more people decide to vaccinate, and thus it is necessary to identify relevant factors involved in vaccination behavior to find better ways of encouraging it. Vaccination behavior is the result of a decision process that might vary according to individual differences in information processing. We investigated the role of cognitive reflection ability and thinking styles in predicting self-reported vaccination behavior against COVID-19. A sample of 274 Romanian participants was surveyed for the present study, out of which 217 (M_age = 24.58, SD = 8.31; 53% female) declared they had the possibility to become vaccinated. Results showed that a higher level of cognitive reflection ability significantly increased the odds of becoming vaccinated. A rational thinking style was not linked to vaccination behavior. However, an experiential thinking style indirectly predicted vaccination behavior by means of attitudes towards vaccination. Since individual differences in information processing are, to a certain extent, linked to vaccination behavior, the design of vaccination campaigns could consider that people have specific information needs and address them as such.

A 21st century cognitive portrait of the Himba, a remote people of Namibia

This research sketches the cognitive portrait of the Himba, a remote population from Northern Namibia living in a non-industrial society almost completely devoid of modern artefacts. We compared the Himba sample to a French sample, exploring cognitive reflection, moral judgement, cooperative behaviour, paranormal beliefs, and happiness. We looked for both differences and similarities across cultures, and for the way cognitive functioning is associated with a range of demographic variables. Results showed some important group differences, with the Himba being more intuitive, more religious, happier, and less utilitarian than the French participants. Further, the predictors of these beliefs and behaviours differed between the two groups. The present results provide additional support to the recent line of research targeting cultural variations and similarities, and call for the need to expand psychology research beyond the Western world.

Criterion Validity of Cognitive Reflection for Predicting Job Performance and Training Proficiency: A Meta-Analysis

This article presents a meta-analysis of the validity of cognitive reflection (CR) for predicting job performance and training proficiency. It also examines the incremental validity of CR over cognitive intelligence (CI) for predicting these two occupational criteria. CR proved to be an excellent predictor of job performance and training proficiency, and the magnitude of the true validity was very similar across the two criteria. Results also showed that the type of CR is not a moderator of CR validity. We also found that CR showed incremental variance over CI for the explanation of job performance, although the magnitude of the contribution is small. However, CR shows practically no incremental validity over CI validity in the explanation of training proficiency. Finally, we discuss the implications of these findings for the research and practice of personnel selection.