How to find what's in a name: Scrutinizing the optimality of five scoring algorithms for the name‐letter task

Although the name‐letter task (NLT) has become an increasingly popular technique to measure implicit self‐esteem (ISE), researchers have relied on different algorithms to compute NLT scores and the psychometric properties of these differently computed scores have never been thoroughly investigated. Based on 18 independent samples, including 2690 participants, the current research examined the optimality of five scoring algorithms based on the following criteria: reliability; variability in reliability estimates across samples; types of systematic error variance controlled for; systematic production of outliers and shape of the distribution of scores. Overall, an ipsatized version of the original algorithm exhibited the most optimal psychometric properties, which is recommended for future research using the NLT. Copyright © 2009 John Wiley & Sons, Ltd.

A Unified Framework to Quantify the Credibility of Scientific Findings

Societies invest in scientific studies to better understand the world and attempt to harness such improved understanding to address pressing societal problems. Published research, however, can be useful for theory or application only if it is credible. In science, a credible finding is one that has repeatedly survived risky falsification attempts. However, state-of-the-art meta-analytic approaches cannot determine the credibility of an effect because they do not account for the extent to which each included study has survived such attempted falsification. To overcome this problem, we outline a unified framework for estimating the credibility of published research by examining four fundamental falsifiability-related dimensions: (a) transparency of the methods and data, (b) reproducibility of the results when the same data-processing and analytic decisions are reapplied, (c) robustness of the results to different data-processing and analytic decisions, and (d) replicability of the effect. This framework includes a standardized workflow in which the degree to which a finding has survived scrutiny is quantified along these four facets of credibility. The framework is demonstrated by applying it to published replications in the psychology literature. Finally, we outline a Web implementation of the framework and conclude by encouraging the community of researchers to contribute to the development and crowdsourcing of this platform.

What Do Implicit Measures Tell Us?: Scrutinizing the Validity of Three Common Assumptions

Experimental paradigms designed to assess “implicit” representations are currently very popular in many areas of psychology. The present article addresses the validity of three widespread assumptions in research using these paradigms: that (a) implicit measures reflect unconscious or introspectively inaccessible representations; (b) the major difference between implicit measures and self-reports is that implicit measures are resistant or less susceptible to social desirability; and (c) implicit measures reflect highly stable, older representations that have their roots in long-term socialization experiences. Drawing on a review of the available evidence, we conclude that the validity of all three assumptions is equivocal and that theoretical interpretations should be adjusted accordingly. We discuss an alternative conceptualization that distinguishes between activation and validation processes.

PsychDisclosure.org

There is currently an unprecedented level of doubt regarding the reliability of research findings in psychology. Many recommendations have been made to improve the current situation. In this article, we report results from PsychDisclosure.org , a novel open-science initiative that provides a platform for authors of recently published articles to disclose four methodological design specification details that are not required to be disclosed under current reporting standards but that are critical for accurate interpretation and evaluation of reported findings. Grassroots sentiment—as manifested in the positive and appreciative response to our initiative—indicates that psychologists want to see changes made at the systemic level regarding disclosure of such methodological details. Almost 50% of contacted researchers disclosed the requested design specifications for the four methodological categories (excluded subjects, nonreported conditions and measures, and sample size determination). Disclosed information provided by participating authors also revealed several instances of questionable editorial practices, which need to be thoroughly examined and redressed. On the basis of these results, we argue that the time is now for mandatory methods disclosure statements for all psychology journals, which would be an important step forward in improving the reliability of findings in psychology.

The Interactive Role of Implicit and Explicit Partner Evaluations on Ongoing Affective and Behavioral Romantic Realities

Past research on close relationships has increasingly focused on the assessment of implicit constructs to shed new light on relationship processes. However, virtually nothing is known about the role of such constructs in understanding ongoing affective and behavioral romantic realities and how implicit and explicit relationship constructs interact in the context of daily relationship outcomes. Using a 21-day diary approach, the present research examined the unique and interactive role of implicit partner evaluations and explicit partner perceptions on relationship outcomes (daily relationship quality and positive relationship behaviors enacted toward partner). Results showed that more positive implicit partner evaluations uniquely predicted more positive relationship outcomes during the 21-day diary period, but that this was especially pronounced in individuals who did not explicitly perceive their partner’s attributes in an overly positive manner. Implications for the close relationship literature are discussed.

Socially desirable responding and its elusive effects on the validity of personality assessments

Past studies of socially desirable self-reports on the items of personality measures have found inconsistent effects of the response bias on the measures' predictive validities, with some studies reporting small effects and other studies reporting large effects. Using Monte Carlo methods, we evaluated various models of socially desirable responding by systematically adding predetermined amounts of the bias to the simulated personality trait scores of hypothetical test respondents before computing test-criterion validity correlations. Our study generally supported previous findings that have reported relatively minor decrements in criterion prediction, even with personality scores that were massively infused with desirability bias. Furthermore, the response bias failed to reveal itself as a statistical moderator of test validity or as a suppressor of validity. Large differences between some respondents' obtained test scores and their true trait scores, however, meant that the personality measure's construct validity would be severely compromised and, more specifically, that estimates of those individuals' criterion performance would be grossly in error. Our discussion focuses on reasons for the discrepant results reported in the literature pertaining to the effect of socially desirable responding on criterion validity. More important, we explain why the lack of effects of desirability bias on the usual indicators of validity, moderation, and suppression should not be surprising.

Name-Letter Preferences for New Last Name and Abandoned Birth Name Initials in the Context of Name-Change via Marriage

Although the name-letter task is one of the most frequently used measures of implicit self-esteem, no research has examined whether the name-letter effect emerges for new last name initials and abandoned birth name initials in the context of marriage. Additionally, no systematic investigation has examined the robustness of the name-letter effect across age cohorts. In a large heterogeneous sample (N = 1,380), reliable letter preferences were found for new last name initials and for abandoned birth name initials, even after 20 years of marriage. In addition, robust name-letter effects emerged across all assessed age cohorts. Implications for the implicit self-esteem literature regarding the robustness of the name-letter task for married and nonmarried individuals of all post-pubescent ages are discussed.

Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice

In this methodological commentary, we use Bem's (2011) recent article reporting experimental evidence for psi as a case study for discussing important deficiencies in modal research practice in empirical psychology. We focus on (a) overemphasis on conceptual rather than close replication, (b) insufficient attention to verifying the soundness of measurement and experimental procedures, and (c) flawed implementation of null hypothesis significance testing. We argue that these deficiencies contribute to weak method-relevant beliefs that, in conjunction with overly strong theory-relevant beliefs, lead to a systemic and pernicious bias in the interpretation of data that favors a researcher's theory. Ultimately, this interpretation bias increases the risk of drawing incorrect conclusions about human psychology. Our analysis points to concrete recommendations for improving research practice in empirical psychology. We recommend (a) a stronger emphasis on close replication, (b) routinely verifying the integrity of measurement instruments and experimental procedures, and (c) using stronger, more diagnostic forms of null hypothesis testing.

Sexy But Often Unreliable: The Impact of Unreliability on the Replicability of Experimental Findings With Implicit Measures

Implicit measures have contributed to important insights in almost every area of psychology. However, various issues and challenges remain concerning their use, one of which is their considerable variation in reliability, with many implicit measures having questionable reliability. The goal of the present investigation was to examine an overlooked consequence of this liability with respect to replication, when such implicit measures are used as dependent variables in experimental studies. Using a Monte Carlo simulation, the authors demonstrate that a higher level of unreliability in such dependent variables is associated with substantially lower levels of replicability. The results imply that this overlooked consequence can have far-reaching repercussions for the development of a cumulative science. The authors recommend the routine assessment and reporting of the reliability of implicit measures and also urge the improvement of implicit measures with low reliability.

Response Interference as a Mechanism Underlying Implicit Measures

Over the last decade, implicit measures of mental associations (e.g., Implicit Association Test, sequential priming) have become increasingly popular in many areas of psychological research. Even though successful applications provide preliminary support for the validity of these measures, their underlying mechanisms are still controversial. The present article addresses the role of a particular mechanism that is hypothesized to mediate the influence of activated associations on task performance in many implicit measures: response interference (RI). Based on a review of relevant evidence, we argue that RI effects in implicit measures depend on participants’ attention to association-relevant stimulus features, which in turn can influence the reliability and the construct validity of these measures. Drawing on a moderated-mediation model (MMM) of task performance in RI paradigms, we provide several suggestions on how to address these problems in research using implicit measures.