The MTurkification of Social and Personality Psychology

Challenging the N-Heuristic: Effect size, not sample size, predicts the replicability of psychological science

Large sample size (N) is seen as a key criterion in judging the replicability of psychological research, a phenomenon we refer to as the N-Heuristic. This heuristic has led to the incentivization of fast, online, non-behavioral studies-to the potential detriment of psychological science. While large N should in principle increase statistical power and thus the replicability of effects, in practice it may not. Large-N studies may have other attributes that undercut their power or validity. Consolidating data from all systematic, large-scale attempts at replication (N = 307 original-replication study pairs), we find that the original study's sample size did not predict its likelihood of being replicated (rs = -0.02, p = 0.741), even with study design and research area controlled. By contrast, effect size emerged as a substantial predictor (rs = 0.21, p < 0.001), which held regardless of the study's sample size. N may be a poor predictor of replicability because studies with larger N investigated smaller effects (rs = -0.49, p < 0.001). Contrary to these results, a survey of 215 professional psychologists, presenting them with a comprehensive list of methodological criteria, found sample size to be rated as the most important criterion in judging a study's replicability. Our findings strike a cautionary note with respect to the prioritization of large N in judging the replicability of psychological science.

Power to Detect What? Considerations for Planning and Evaluating Sample Size

ACADEMIC ABSTRACT

In the wake of the replication crisis, social and personality psychologists have increased attention to power analysis and the adequacy of sample sizes. In this article, we analyze current controversies in this area, including choosing effect sizes, why and whether power analyses should be conducted on already-collected data, how to mitigate the negative effects of sample size criteria on specific kinds of research, and which power criterion to use. For novel research questions, we advocate that researchers base sample sizes on effects that are likely to be cost-effective for other people to implement (in applied settings) or to study (in basic research settings), given the limitations of interest-based minimums or field-wide effect sizes. We discuss two alternatives to power analysis, precision analysis and sequential analysis, and end with recommendations for improving the practices of researchers, reviewers, and journal editors in social-personality psychology.

PUBLIC ABSTRACT

Recently, social-personality psychology has been criticized for basing some of its conclusions on studies with low numbers of participants. As a result, power analysis, a mathematical way to ensure that a study has enough participants to reliably "detect" a given size of psychological effect, has become popular. This article describes power analysis and discusses some controversies about it, including how researchers should derive assumptions about effect size, and how the requirements of power analysis can be applied without harming research on hard-to-reach and marginalized communities. For novel research questions, we advocate that researchers base sample sizes on effects that are likely to be cost-effective for other people to implement (in applied settings) or to study (in basic research settings). We discuss two alternatives to power analysis, precision analysis and sequential analysis, and end with recommendations for improving the practices of researchers, reviewers, and journal editors in social-personality psychology.

Natural clues to danger: attachment behavior in threatening situations

A fundamental principle of attachment theory is that threatening situations give rise to individual differences in the extent to which people seek proximity to close others. The current research examines the way in which attachment styles predict individual differences in attachment-relevant behavior during threatening events. We tested alternative theoretical perspectives concerning the association between adult attachment (specifically, attachment avoidance) and attachment behavior in the presence of natural clues to danger by observing couples (N = 204) when they were watching horror vs. control film excerpts. Results suggest that highly avoidant people engaged in less attachment behavior across both threatening and non-threatening situations. These findings have implications for the understanding of attachment-related processes and how working models of the self and others facilitate (or inhibit) the expression of attachment behavior.

The Burden for High-Quality Online Data Collection Lies With Researchers, Not Recruitment Platforms

A recent article in Perspectives on Psychological Science (Webb & Tangney, 2022) reported a study in which just 2.6% of participants recruited on Amazon's Mechanical Turk (MTurk) were deemed "valid." The authors highlighted some well-established limitations of MTurk, but their central claims-that MTurk is "too good to be true" and that it captured "only 14 human beings . . . [out of] N = 529"-are radically misleading, yet have been repeated widely. This commentary aims to (a) correct the record (i.e., by showing that Webb and Tangney's approach to data collection led to unusually low data quality) and (b) offer a shift in perspective for running high-quality studies online. Negative attitudes toward MTurk sometimes reflect a fundamental misunderstanding of what the platform offers and how it should be used in research. Beyond pointing to research that details strategies for effective design and recruitment on MTurk, we stress that MTurk is not suitable for every study. Effective use requires specific expertise and design considerations. Like all tools used in research-from advanced hardware to specialist software-the tool itself places constraints on what one should use it for. Ultimately, high-quality data is the responsibility of the researcher, not the crowdsourcing platform.

Too Anecdotal to Be True? Mechanical Turk Is Not All Bots and Bad Data: Response to Webb and Tangney (2022)

In response to Webb and Tangney (2022) we call into question the conclusion that data collected on Amazon's Mechanical Turk (MTurk) was "at best-only 2.6% valid" (p. 1). We suggest that Webb and Tangney made certain choices during the study-design and data-collection process that adversely affected the quality of the data collected. As a result, the anecdotal experience of these authors provides weak evidence that MTurk provides low-quality data as implied. In our commentary we highlight best practice recommendations and make suggestions for more effectively collecting and screening online panel data.

Artificial intelligence and illusions of understanding in scientific research

Scientists are enthusiastically imagining ways in which artificial intelligence (AI) tools might improve research. Why are AI tools so attractive and what are the risks of implementing them across the research pipeline? Here we develop a taxonomy of scientists' visions for AI, observing that their appeal comes from promises to improve productivity and objectivity by overcoming human shortcomings. But proposed AI solutions can also exploit our cognitive limitations, making us vulnerable to illusions of understanding in which we believe we understand more about the world than we actually do. Such illusions obscure the scientific community's ability to see the formation of scientific monocultures, in which some types of methods, questions and viewpoints come to dominate alternative approaches, making science less innovative and more vulnerable to errors. The proliferation of AI tools in science risks introducing a phase of scientific enquiry in which we produce more but understand less. By analysing the appeal of these tools, we provide a framework for advancing discussions of responsible knowledge production in the age of AI.

Comparing data quality from an online and in-person lab sample on dynamic theory of mind tasks

Nearly half the published research in psychology is conducted with online samples, but the preponderance of these studies rely primarily on self-report measures. The current study validated data quality from an online sample on a novel, dynamic task by comparing performance between an in-lab and online sample on two dynamic measures of theory of mind-the ability to infer others' mental states. Theory of mind is a cognitively complex construct that has been widely studied across multiple domains of psychology. One task was based on the show The Office®, and has been previously validated by the authors with in-lab samples. The second was a novel task based on the show Nathan for You®, which was selected to account for familiarity effects associated with The Office. Both tasks measured various dimensions of theory of mind (inferring beliefs, understanding motivations, detecting deception, identifying faux pas, and understanding emotions). The in-person lab samples (N = 144 and 177, respectively) completed the tasks between-subject, whereas the online sample (N = 347 from Prolific Academic) completed them within-subject, with order counterbalanced. The online sample's performance across both tasks was reliable (Cronbach's α = .66). For The Office, the in-person sample outperformed the online sample on some types of theory of mind, but this was driven by their greater familiarity with the show. Indeed, for the relatively unfamiliar show Nathan for You, performance did not differ between the two samples. Together, these results suggest that crowdsourcing platforms elicit reliable performance on novel, dynamic, complex tasks.

Is it ethical to use Mechanical Turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages

To understand human behavior, social scientists need people and data. In the last decade, Amazon's Mechanical Turk (MTurk) emerged as a flexible, affordable, and reliable source of human participants and was widely adopted by academics. Yet despite MTurk's utility, some have questioned whether researchers should continue using the platform on ethical grounds. The brunt of their concern is that people on MTurk are financially insecure, subject to abuse, and earn inhumane wages. We investigated these issues with two representative probability surveys of the U.S. MTurk population (N = 4094). The surveys revealed: (1) the financial situation of people on MTurk mirrors the general population, (2) most participants do not find MTurk stressful or requesters abusive, and (3) MTurk offers flexibility and benefits that most people value above other options for work. People reported it is possible to earn more than $10 per hour and said they would not trade the flexibility of MTurk for less than $25 per hour. Altogether, our data are important for assessing whether MTurk is an ethical place for research.

Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory

Terror Management Theory (TMT) postulates that humans, in response to awareness of their death, developed complex defenses to remove the salience and discomfort stemming from those thoughts. In a standard paradigm to test this theory, an individual is presented with a death-related prime (Mortality Salience; MS), such as writing the details of their own death, or something neutral, such as watching television. After a distractor task (for delay), participants complete the dependent variable, such as rating how much they like or agree with a pro- or anti-national essay and its author. Individuals in the MS condition typically exhibit greater worldview defense than control conditions by rating the pro-national essay more positively and the anti-national essay more negatively. We completed five separate studies across five unique samples with the goal of replicating and extending this well-established pattern to provide further understanding of the phenomena that underlie the effects of MS. However, despite using standard procedures, we were unable to replicate basic patterns of the dependent variable in the MS conditions. We also pooled all responses into two meta-analyses, one examining all dependent variables and one focusing on the anti-national essay; yet the effect sizes in these analyses did not significantly differ from zero. We discuss the methodological and theoretical implications of these (unintended) failures to replicate. It is not clear if these null findings were due to methodological limitations, restraints of online/crowd-sourced recruitment, or ever-evolving sociocultural factors.

Rating scales institutionalise a network of logical errors and conceptual problems in research practices: A rigorous analysis showing ways to tackle psychology's crises

This article explores in-depth the metatheoretical and methodological foundations on which rating scales-by their very conception, design and application-are built and traces their historical origins. It brings together independent lines of critique from different scholars and disciplines to map out the problem landscape, which centres on the failed distinction between psychology's study phenomena (e.g., experiences, everyday constructs) and the means of their exploration (e.g., terms, data, scientific constructs)-psychologists' cardinal error. Rigorous analyses reveal a dense network of 12 complexes of problematic concepts, misconceived assumptions and fallacies that support each other, making it difficult to be identified and recognised by those (unwittingly) relying on them (e.g., various forms of reductionism, logical errors of operationalism, constructification, naïve use of language, quantificationism, statisticism, result-based data generation, misconceived nomotheticism). Through the popularity of rating scales for efficient quantitative data generation, uncritically interpreted as psychological measurement, these problems have become institutionalised in a wide range of research practices and perpetuate psychology's crises (e.g., replication, confidence, validation, generalizability). The article provides an in-depth understanding that is needed to get to the root of these problems, which preclude not just measurement but also the scientific exploration of psychology's study phenomena and thus its development as a science. From each of the 12 problem complexes; specific theoretical concepts, methodologies and methods are derived as well as key directions of development. The analyses-based on three central axioms for transdisciplinary research on individuals, (1) complexity, (2) complementarity and (3) anthropogenicity-highlight that psychologists must (further) develop an explicit metatheory and unambiguous terminology as well as concepts and theories that conceive individuals as living beings, open self-organising systems with complementary phenomena and dynamic interrelations across their multi-layered systemic contexts-thus, theories not simply of elemental properties and structures but of processes, relations, dynamicity, subjectivity, emergence, catalysis and transformation. Philosophical and theoretical foundations of approaches suited for exploring these phenomena must be developed together with methods of data generation and methods of data analysis that are appropriately adapted to the peculiarities of psychologists' study phenomena (e.g., intra-individual variation, momentariness, contextuality). Psychology can profit greatly from its unique position at the intersection of many other disciplines and can learn from their advancements to develop research practices that are suited to tackle its crises holistically.

Methods and measures in social and personality psychology: a comparison of JPSP publications in 1982 and 2016

Reviews of articles from social and personality psychology journals have been largely limited to subsets of publications from particular years and to a focus on descriptive qualities of articles. This paper compares the methods and measures employed in all empirical articles published in 1982 to those that appeared in 2016 from Journal of Personality and Social Psychology. There has been an increase in self-report data, a decline in behavioral observations, and an increase in detailed reports of procedures and analyses. A heavy reliance on laboratory experimentation involving college students has been supplanted by online participant pools and data collection procedures. Compared to 1982, articles in 2016 were fewer in number but longer in length, included more studies per article, and had a greater number and diversity of authors. Explanations are offered for these findings along with implications for social and personality psychology.

Scale Mean and Variance Differences in MTurk and Non-MTurk Samples

Abstract. We meta-analytically examined mean and variance differences between MTurk and non-MTurk samples for a variety of scales used in the organizational sciences. The influence of moderators (i.e., construct domain and valence, sample type, use of qualifications, and data cleaning procedures) was also examined. Across all scales (120 scales, N = 110,090), we found that, overall, MTurk and non-MTurk samples do not have significantly different scale means or variances. Our moderator analyses, however, indicated that MTurk samples may have larger variances than non-MTurk samples under certain conditions. Additionally, our results highlight the importance of data screening and cleaning procedures such that MTurk samples had larger variances than non-MTurk samples when MTurk samples were subjected to more stringent data screening and cleaning procedures.

Moral psychology from the lab to the wild: Relief registries as a paradigm for studying real-world altruism

Experimental psychology's recent shift toward low-effort, high-volume methods (e.g., self-reports, online studies) and away from the more effortful study of naturalistic behavior raises concerns about the ecological validity of findings from these fields, concerns that have become particularly apparent in the field of moral psychology. To help address these concerns, we introduce a method allowing researchers to investigate an important, widespread form of altruistic behavior-charitable donations-in a manner balancing competing concerns about internal validity, ecological validity, and ease of implementation: relief registries, which leverage existing online gift registry platforms to allow research subjects to choose among highly needed donation items to ship directly to charitable organizations. Here, we demonstrate the use of relief registries in two experiments exploring the ecological validity of the finding from our own research that people are more willing to help others after having imagined themselves doing so. In this way, we sought to provide a blueprint for researchers seeking to enhance the ecological validity of their own research in a narrow sense (i.e., by using the relief registry method we introduce) and in broader terms by adapting methods that take advantage of modern technology to directly impact others' lives outside the lab.

The Pandemic as a Portal: Reimagining Psychological Science as Truly Open and Inclusive

Psychological science is at an inflection point: The COVID-19 pandemic has exacerbated inequalities that stem from our historically closed and exclusive culture. Meanwhile, reform efforts to change the future of our science are too narrow in focus to fully succeed. In this article, we call on psychological scientists-focusing specifically on those who use quantitative methods in the United States as one context for such conversations-to begin reimagining our discipline as fundamentally open and inclusive. First, we discuss whom our discipline was designed to serve and how this history produced the inequitable reward and support systems we see today. Second, we highlight how current institutional responses to address worsening inequalities are inadequate, as well as how our disciplinary perspective may both help and hinder our ability to craft effective solutions. Third, we take a hard look in the mirror at the disconnect between what we ostensibly value as a field and what we actually practice. Fourth and finally, we lead readers through a roadmap for reimagining psychological science in whatever roles and spaces they occupy, from an informal discussion group in a department to a formal strategic planning retreat at a scientific society.

How Cross-Discipline Understanding and Communication Can Improve Research on Multiracial Populations

One of the strengths of Critical Mixed Race Studies is that it represents research methodologies and frameworks from multiple disciplines across the social sciences and humanities. However, if these disciplines are not in dialogue with each other, that benefit may be lost. Here, we use psychological and sociological research on Multiracial populations as examples to argue how strict disciplinarity and methodological trends may limit scientific production. We propose that reading and citing work across disciplines, expanding methodological training, and rejecting hegemonic “white logic” assumptions about what is “publishable” can enhance Multiracial research. First, the ability to cite effectively across disciplines will shorten the time it takes for new theories to be developed that focus on empirically underrepresented populations. Secondly, increasing understanding of both quantitative and qualitative methods will allow more effective reading between disciplines while also creating opportunities to engage with both causality and the richness of experiences that comprise being Multiracial. Finally, these changes would then situate scholars to be more effective reviewers, thereby enhancing the peer-reviewed publication process to one that routinely rejects color evasive racist practices that privilege work on majority populations.

Buying to Cope With Scarcity During Public Emergencies: A Serial Mediation Model Based on Cognition-Affect Theory

Panic buying is a common phenomenon that occurs during public emergencies and has a significant undesirable impact on society. This research explored the effect of scarcity on panic buying and the role of perceived control and panic in this effect through big data, an online survey and behavior experiments in a real public emergency (i.e., COVID-19) and simulative public emergencies. The findings showed that scarcity aggravates panic buying (Studies 1–3), and this aggravation effect is serially mediated by perceived control and panic (Studies 2–3). Moreover, this serial mediation model is more suitable for public health emergencies (Study 3). These findings enrich the understanding of panic buying and provide important enlightenment for guiding rational public behavior and managing public opinion during public emergencies.

International psychological research addressing the early phase of the COVID-19 pandemic: A rapid scoping review and implications for global psychology

In March 2020, the World Health Organization (WHO) declared COVID-19 a global pandemic. Given that keeping abreast of international perspectives and research results is of particular importance for such massive global emergencies, we employed a scoping review methodology to rapidly map the field of international psychological research addressing this important early phase of the pandemic. We included a total of 79 studies, with data mostly collected between March and June 2020. This review aimed to systematically identify and map the nature and scope of international studies examining psychological aspects of the unfolding COVID-19 pandemic. We mapped key research themes, subfields of psychology, the nature and extent of international research collaboration, data methods employed, and challenges and enablers faced by psychological researchers in the early stages of the pandemic. Among the wide range of themes covered, mental health and social behaviours were the key themes. Most studies were in clinical/health psychology and social psychology. Network analyses revealed how authors collaborated and to what extent the studies were international. Europe and the United States were often at the centre of international collaboration. The predominant study design was cross-sectional and online with quantitative analyses. We also summarised author reported critical challenges and enablers for international psychological research during the COVID pandemic, and conclude with implications for the field of psychology.

Web-Based Research in Psychology

Abstract. The present article reviews web-based research in psychology. It captures principles, learnings, and trends in several types of web-based research that show similar developments related to web technology and its major shifts (e.g., appearance of search engines, browser wars, deep web, commercialization, web services, HTML5…) as well as distinct challenges. The types of web-based research discussed are web surveys and questionnaire research, web-based tests, web experiments, Mobile Experience Sampling, and non-reactive web research, including big data. A number of web-based methods are presented and discussed that turned out to become important in research methodology. These are one-item-one-screen design, seriousness check, instruction manipulation and other attention checks, multiple site entry technique, subsampling technique, warm-up technique, and web-based measurement. Pitfalls and best practices are described then, especially regarding dropout and other non-response, recruitment of participants, and interaction between technology and psychological factors. The review concludes with a discussion of important concepts that have developed over 25 years and an outlook on future developments in web-based research.

On the (Mis)Use of Deception in Web-Based Research

Abstract. The deception of research participants remains a controversial issue in the behavioral sciences. Current ethics codes consistently limit the use of deception to cases in which non-deceptive alternatives are unfeasible and, crucially, require that participants subjected to deception be debriefed correspondingly along with an option to withdraw their data after learning about the deception. These conditions pose a particular challenge in the context of web-based research because participants can typically discontinue a study unilaterally (i.e., dropout by simply closing the browser window) in which case full debriefing and an option to withdraw one’s data are no longer available. As a consequence, the study would no longer be compatible with ethical standards. Based on recent meta-analytical data, we provide an existence proof of this problem, showing that deception is used in web-based research with little to no indication of safeguards ensuring full debriefing and subsequent data withdrawal options. We close by revisiting recommendations for the (non-)use of deception in web-based research and offer solutions to implement such safeguards in case deception is truly unavoidable.

Moderated Online Data-Collection for Developmental Research: Methods and Replications

Online data collection methods are expanding the ease and access of developmental research for researchers and participants alike. While its popularity among developmental scientists has soared during the COVID-19 pandemic, its potential goes beyond just a means for safe, socially distanced data collection. In particular, advances in video conferencing software has enabled researchers to engage in face-to-face interactions with participants from nearly any location at any time. Due to the novelty of these methods, however, many researchers still remain uncertain about the differences in available approaches as well as the validity of online methods more broadly. In this article, we aim to address both issues with a focus on moderated (synchronous) data collected using video-conferencing software (e.g., Zoom). First, we review existing approaches for designing and executing moderated online studies with young children. We also present concrete examples of studies that implemented choice and verbal measures (Studies 1 and 2) and looking time (Studies 3 and 4) across both in-person and online moderated data collection methods. Direct comparison of the two methods within each study as well as a meta-analysis of all studies suggest that the results from the two methods are comparable, providing empirical support for the validity of moderated online data collection. Finally, we discuss current limitations of online data collection and possible solutions, as well as its potential to increase the accessibility, diversity, and replicability of developmental science.

Suspicious and fraudulent online survey participation: Introducing the REAL framework

Online survey research has significantly increased in popularity in recent years. With its use, researchers have a new set of concerns about data collection and analysis to consider, including the possibility of fraudulent survey submissions. The purpose of this article is to demonstrate to survey researchers an innovative and systematized process for addressing online survey fraud over the course of collecting survey data, especially when respondents collect incentives for participation. We provide the Reflect, Expect, Analyze, Label Framework, which includes four sets of guiding questions for use by online survey researchers to plan for addressing survey fraud and making determinations about the inclusion or exclusion of participant submissions from the dataset based on level of suspicion. We also provide a full case example utilizing the Reflect, Expect, Analyze, Label Framework as an appendix. Those wanting to apply the Reflect, Expect, Analyze, Label Framework should keep in mind several considerations as they apply it, including determining logistical needs ahead of survey implementation, considering the ethical issues related to including or excluding data in a study, and considering the issues related to providing incentives for participating in research. Future research should assess the frequency of survey fraud, investigate the reasons for its occurrence and explore the role social networks may play in fraudulent participants sharing information. We suggest that researchers consider online survey fraud as an issue over the lifespan of their survey and apply the guiding questions we present to address the issue throughout.

Searching for meaning is associated with costly prosociality

The study of meaning in life has largely centered on its relationship with personal well-being, while a focus on how meaning is related to enhancing the well-being of others has received less research attention. Although searching for meaning may imply lower personal well-being, we find that meaning-seekers are more motivated to perform costly prosocial actions for the sake of others’ well-being, given the perceived meaningfulness of these behaviors. Studies 1–4 (N = 780) show that meaning-seeking correlates with the motivation to engage in a range of costly prosocial behaviors. Meaning-seeking is further shown to be distinct from pursuing happiness in its relationship with costly prosociality (Study 2 & 3) and to share a stronger association with high-cost than low-cost prosociality (Study 3 & 4). Study 5 (N = 370; pre-registered) further shows that the search for meaning is related to costly prosocial behavior in the recent past. While our studies are cross-sectional, the pattern of findings suggests that seeking meaning (rather than happiness) may play an important role in motivating altruistic tendencies.

Narrative identity informs psychological adjustment: Considering three themes captured across five time points and two event valences

OBJECTIVES

How narrative identity and well-being are intertwined as emerging adults process their lived experiences remains a critical theoretical and empirical question. We studied narrative identity among US emerging adults in a multiphase study. We aimed to test (1) if and how narrative identity themes (i.e., coherence, agency, growth) change rapidly across repeated narrations; (2) are related to reports of psychological adjustment (i.e., well-being, recent stress) over time; and (3) whether the valence of the autobiographical event nuanced the ways narrative identity and adjustment co-evolve.

METHODS

In a mini-longitudinal study conducted over three months, 300 adults aged 18-to-29-years (M age = 24.39 years; 60% women) provided autobiographical narratives about high-point and low-point (LP) life events at five time points, as well as repeated reports on well-being and recent stress.

RESULTS

Overall, coherence showed (1) the most consistency across time and valence than other narrative themes and (2) the most consistent associations with adjustment. In multilevel models, LP coherence and LP growth coincided with higher levels of adjustment.

CONCLUSIONS

Findings reinforce the ways narrative identity reflects dynamic processes of understanding the events of one's life, and the ways individual differences in framing and reasoning about life are important for psychological adjustment.

Evaluating Research in Personality and Social Psychology: Considerations of Statistical Power and Concerns About False Findings

Traditionally, statistical power was viewed as relevant to research planning but not evaluation of completed research. However, following discussions of high false finding rates (FFRs) associated with low statistical power, the assumed level of statistical power has become a key criterion for research acceptability. Yet, the links between power and false findings are not as straightforward as described. Assumptions underlying FFR calculations do not reflect research realities in personality and social psychology. Even granting the assumptions, the FFR calculations identify important limitations to any general influences of statistical power. Limits for statistical power in inflating false findings can also be illustrated through the use of FFR calculations to (a) update beliefs about the null or alternative hypothesis and (b) assess the relative support for the null versus alternative hypothesis when evaluating a set of studies. Taken together, statistical power should be de-emphasized in comparison to current uses in research evaluation.

Development of a Simplified Portion Size Selection Task

Portion size is an important determinant of energy intake and the development of easy to use and valid tools for measuring portion size are required. Standard measures, such as ad libitum designs and currently available computerized portion selection tasks (PSTs), have several limitations including only being able to capture responses to a limited number of foods, requiring participants’ physical presence and logistical/technical demands. The objective of the current study was to develop and test robust and valid measures of portion size that can be readily prepared by researchers and be reliably utilized for remote online data collection. We developed and tested two simplified PSTs that could be utilized online: (1) portion size images presented simultaneously along a horizontal continuum slider and (2) multiple-choice images presented vertically. One hundred and fifty participants (M = 21.35 years old) completed both simplified PSTs, a standard computerized PST and a series of questionnaires of variables associated with portion size (e.g., hunger, food item characteristics, Three Factor Eating Questionnaire). We found average liking of foods was a significant predictor of all three tasks and cognitive restraint also predicted the two simplified PSTs. We also found significant agreement between the standard PST and estimated portion sizes derived from the simplified PSTs when accounting for average liking. Overall, we show that simplified versions of the standard PST can be used online as an analogue of estimating ideal portion size.

Adults' attitudes toward children, adolescents, and adults who are obese and tax-funded programs to combat obesity

The goals of the present study were to assess (1) adults' patterns of beliefs about and attitudes toward children, adolescents, and adults who are obese and (2) their attitudes toward hypothetical tax-funded programs designed to combat obesity in children, adolescents, and adults. A total of 267 participants, ranging in age from 19 to 88 years old, were recruited through Amazon's Mechanical Turk (MTurk) to participate online in the present study. The participants rated adults who are obese less favorably, and as less malleable, than children or adolescents who are obese. Furthermore, they were less supportive of tax-funded programs designed to combat obesity in adults than children or adolescents. In general, the participants' relatively unfavorable response to adults who are obese appears to be associated with the beliefs that older individuals who are obese are relatively unchangeable and have heightened personal fault for their plight.

Extracting Agency and Communion From the Big Five: A Four-Way Competition

Agency and communion are the two fundamental content dimensions in psychology. The two dimensions figure prominently in many psychological realms (personality, social, self, motivational, cross-cultural, etc.). In contemporary research, however, personality is most commonly measured within the Big Five framework. We developed novel agency and communion scales based on the items from the most popular nonpropriety measure of the Big Five-the Big Five Inventory. We compared four alternative scale-construction methods: expert rating, target scale, ant colony, and brute force. Across three samples (N_total = 942), all methods yielded reliable and valid agency and communion scales. Our research provides two main contributions. For psychometric theory, it extends knowledge on the four scale-construction methods and their relative convergence. For psychometric practice, it enables researchers to examine agency and communion hypotheses with extant Big Five Inventory data sets, including those collected in their own labs as well as openly accessible, large-scale data sets.

A Validity-Based Framework for Understanding Replication in Psychology

In recent years, psychology has wrestled with the broader implications of disappointing rates of replication of previously demonstrated effects. This article proposes that many aspects of this pattern of results can be understood within the classic framework of four proposed forms of validity: statistical conclusion validity, internal validity, construct validity, and external validity. The article explains the conceptual logic for how differences in each type of validity across an original study and a subsequent replication attempt can lead to replication “failure.” Existing themes in the replication literature related to each type of validity are also highlighted. Furthermore, empirical evidence is considered for the role of each type of validity in non-replication. The article concludes with a discussion of broader implications of this classic validity framework for improving replication rates in psychological research.

Field experiment in psychology. Is there still room for conducting research in natural conditions?

Background

This paper reflects on the status of the field experiment in contemporary psychology (especially its social branch). The subject of analysis is how frequently this method is used, especially in the context of the growing popularity of on-line research.

Participants and procedure

The empirical part of the article presents an experiment revealing the uniqueness of the field experiment as a method allowing for the identification of subtle variables that can have a real impact on the results obtained. In the whole study 128 people were examined (half of the group were women). The obtained results show that some phenomena cannot be examined in any other way than by means of the field experiment.

Results

The process of moving away from the study of real behaviours in the real environment, observed over the last twenty years, has not stopped since the criticism of this phenomenon expressed about 20 years ago. On the contrary, it has intensified. The phenomenon described in the article is definitely harmful to the scientific disci-pline so strongly connected with the social context of research, namely social psychology.

Conclusions

Time will show whether the increasingly loud voices criticising this trend will translate into an actual change of methods used by social psychologists.

Unburdening the Shoulders of Giants: A Quest for Disconnected Academic Psychology

In current academic psychology, scholars typically develop their research and ideas by drawing on the work of other contemporary and preceding psychological scientists and by following certain conventions of the field. I refer to this variant of psychology as connected because the emphasis is on connecting various research findings and ideas generated by different scholars (e.g., by showing how they are related to each other via referencing). In this article, I argue that, although connected psychology advances psychological knowledge, it restricts the total amount of knowledge that could eventually be produced and therefore limits the potential of the discipline to improve the understanding of psychological phenomena. As a solution, I propose that, alongside the currently existing connected psychology, disconnected psychology should be established. In disconnected psychology, researchers develop their ideas by following the main principles of psychological method, but they are disconnected from a “field” consisting of other psychologists and therefore do not follow the discipline’s norms and conventions. By drawing on one of the core constructs from information theory—information entropy—I argue that combining the two streams of psychology would result in the most significant advancement of psychological knowledge.

The contagious impact of playing violent video games on aggression: Longitudinal evidence

Meta-analyses have shown that violent video game play increases aggression in the player. The present research suggests that violent video game play also affects individuals with whom the player is connected. A longitudinal study (N = 980) asked participants to report on their amount of violent video game play and level of aggression as well as how they perceive their friends and examined the association between the participant's aggression and their friends' amount of violent video game play. As hypothesized, friends' amount of violent video game play at Time 1 was associated with the participant's aggression at Time 2 even when controlling for the impact of the participant's aggression at Time 1. Mediation analyses showed that friends' aggression at Time 1 accounted for the impact of friends' amount of violent video game play at Time 1 on the participant's aggression at Time 2. These findings suggest that increased aggression in video game players has an impact on the player's social network.

Research in Social Psychology Changed Between 2011 and 2016: Larger Sample Sizes, More Self-Report Measures, and More Online Studies

The debate about false positives in psychological research has led to a demand for higher statistical power. To meet this demand, researchers need to collect data from larger samples—which is important to increase replicability, but can be costly in both time and money (i.e., remuneration of participants). Given that researchers might need to compensate for these higher costs, we hypothesized that larger sample sizes might have been accompanied by more frequent use of less costly research methods (i.e., online data collection and self-report measures). To test this idea, we analyzed social psychology studies published in 2009, 2011, 2016, and 2018. Indeed, research reported in 2016 and 2018 (vs. 2009 and 2011) had larger sample sizes and relied more on online data collection and self-report measures. Thus, over these years, research improved in its statistical power, but also changed with regard to the methods applied. Implications for social psychology as a discipline are discussed.

MTurk Participants Have Substantially Lower Evaluative Subjective Well-Being Than Other Survey Participants

Amazon's MTurk platform has become a popular site for obtaining relatively inexpensive and convenient adult samples for use in behavioral research. Concerns have been raised about selection issues, because MTurk workers chose to participate in the platform and select the tasks they perform (of many offered to them). Prior studies have documented demographic and psychological differences with national samples. In this paper we studied evaluative subjective well-being (the Cantril Ladder) in an MTurk sample, a national Internet panel sample, and a national telephone survey conducted by Gallup-Sharecare. A surprising finding was that MTurk participants' Ladder scores were substantial lower than the other two samples. Analyses controlling for six demographic differences among the samples only slightly reduced the mean differences. However, patterns of demographic-well-being associations were similar within the samples. To corroborate these results, we conducted a secondary analysis on another three samples, one MTurk sample and two Internet panel samples. The same group differences in Ladder scores were observed. These findings add to the growing literature documenting the characteristics of MTurk samples and we discuss the implications for future research with such samples.