Practical Methodological Reform Needs Good Theory

Predicted as observed? How to identify empirically adequate theoretical constructs

The identification of an empirically adequate theoretical construct requires determining whether a theoretically predicted effect is sufficiently similar to an observed effect. To this end, we propose a simple similarity measure, describe its application in different research designs, and use computer simulations to estimate the necessary sample size for a given observed effect. As our main example, we apply this measure to recent meta-analytical research on precognition. Results suggest that the evidential basis is too weak for a predicted precognition effect of d = 0.20 to be considered empirically adequate. As additional examples, we apply this measure to object-level experimental data from dissonance theory and a recent crowdsourcing hypothesis test, as well as to meta-analytical data on the correlation of personality traits and life outcomes.

What Can Educational Psychology Learn From, and Contribute to, Theory Development Scholarship?

One of the field of psychology’s stated goals is to produce scholarship with findings that benefit the world. Over the last 10 years, psychology scholarship and its presumed societal benefits have been called into question due to the field’s history of questionable research practices, racism, and epistemic oppression. Calls for methodological, ethical, and practical reforms are essential to building a psychological science that is just, effective, reliable, and beneficial. Recently, these calls have been complemented by a push to reform how theory is developed and used in psychology. There is great need for better understanding and implementation of theory development, both in the field of psychology broadly as well as in its subdisciplines, including educational psychology. At the same time, educational psychologists have much to contribute to the tripartite calls for methodological, ethical, and theory reforms. In this paper, I explore theory development in psychology by discussing the importance of developing both the descriptive and explanatory aspects of theory as well as the need to refine how theory is evaluated and integrated. Then, I review how innovations and practices in educational psychology can inform the broader theory reform movement in psychology writ large. Finally, I identify important future directions for further advancing theory reform in psychology, including addressing ableism in the field, increasing the field’s hospitability for theory revision, furthering theory development transparency, and more deeply integrating theory development into preparation programs.

Mechanistic modeling for the masses

The generalizability crisis is compounded, or even partially caused, by a lack of specificity in psychological theories. Expanding the use of mechanistic models among psychologists is therefore important, but faces numerous hurdles. A cultural evolutionary approach can help guide and evaluate interventions to improve modeling efforts in psychology, such as developing standards and implementing them at the institutional level.

Beyond the "Third Wave of Positive Psychology": Challenges and Opportunities for Future Research

The positive psychology (PP) landscape is changing, and its initial identity is being challenged. Moving beyond the “third wave of PP,” two roads for future research and practice in well-being studies are discerned: The first is the state of the art PP trajectory that will (for the near future) continue as a scientific (sub)discipline in/next to psychology (because of its popular brand name). The second trajectory (main focus of this manuscript) links to pointers described as part of the so-called third wave of PP, which will be argued as actually being the beginning of a new domain of inter- or transdisciplinary well-being studies in its own right. It has a broader scope than the state of the art in PP, but is more delineated than in planetary well-being studies. It is in particular suitable to understand the complex nature of bio-psycho-social-ecological well-being, and to promote health and wellness in times of enormous challenges and changes. A unique cohering focus for this post-disciplinary well-being research domain is proposed. In both trajectories, future research will have to increase cognizance of metatheoretical assumptions, develop more encompassing theories to bridge the conceptual fragmentation in the field, and implement methodological reforms, while keeping context and the interwovenness of the various levels of the scientific text in mind. Opportunities are indicated to contribute to the discourse on the identity and development of scientific knowledge in mainstream positive psychology and the evolving post-disciplinary domain of well-being studies.

The Function of Literature in Psychological Science

The recent reform debates in psychological science, prompted by a widespread crisis of confidence, have exposed and destabilized the so-called myth of self-correction, that is, the problem that most scientists perceive their disciplines as self-correcting without engaging in actual practices that correct the scientific record. In this paper, building on the idea of self-correction as a myth, I propose another myth common to psychological science: the myth of self-organization. The myth of self-organization is the idea that scientific literature will organize itself into something the community adding to it would recognize as systematic knowledge; while the actual members of those communities do not engage in effective ways of organizing it. I argue for the existence of the myth self-organization by taking a historical look at how the scientific literature was construed by psychologists during the 20th century. In my view, the literature, and behaviors of scientists related to it, becomes a social institution exerting influence over the science it belongs to. I conclude with a critical discussion of self-organization through the debates about preregistration and theory formalization in psychology’s reform movement.

How to True Psychology’s Objects

Psychology’s current crisis attends most visibly to perceived problems with statistical models, methods, publication practices, and career incentives. Rarely is close attention given to the objects of inquiry—to ontological matters—yet the crisis-related literature does features statements about the nature of psychology’s objects. Close analysis of the ontological claims reveals discrepant understandings: some researchers assume objects to be stable and singular while others posit them to be dynamic and complex. Nevertheless, both views presume the objects under scrutiny to be real. The analysis also finds each of these ontological claims to be associated not only with particular method prescriptions but also with distinct notions of the scientific self. Though both take the scientific self to be objective, one figures the scientist as not always a rational actor and, therefore, requiring some behavior regulation, while the other sees the scientist as largely capable of self-governing sustained through painstakingly acquired expertise and self-control. The fate of these prevalent assemblages of object, method, and scientific self remains to be determined, yet as conditions of possibility they portend quite different futures. Following description of the assemblages, the article ventures a futuristic portrayal of the scientific practices they each might engender.

Prosociality in science

Science is unthinkable without collaboration between scientists. Yet, science is also unthinkable without competition (i.e., competing for the best and most solid arguments and limited, precious resources). In this review, we argue that scientific work routines represent social dilemmas and that two facets of prosociality help researchers solve these dilemmas: (i) sacrificing personal profit for the sake of collective profit (i.e., cooperation) and (ii) deciding to make oneself vulnerable to exploitation (i.e., trust). We use two contemporary developments in science to illustrate our reasoning: First, researchers' willingness to engage with the lay public (e.g., investing one's limited time to public engagement) and second, their commitment to 'open science' (e.g., sharing one's data and materials despite the risk of exploitation).

The case for formal methodology in scientific reform

Current attempts at methodological reform in sciences come in response to an overall lack of rigor in methodological and scientific practices in experimental sciences. However, most methodological reform attempts suffer from similar mistakes and over-generalizations to the ones they aim to address. We argue that this can be attributed in part to lack of formalism and first principles. Considering the costs of allowing false claims to become canonized, we argue for formal statistical rigor and scientific nuance in methodological reform. To attain this rigor and nuance, we propose a five-step formal approach for solving methodological problems. To illustrate the use and benefits of such formalism, we present a formal statistical analysis of three popular claims in the metascientific literature: (i) that reproducibility is the cornerstone of science; (ii) that data must not be used twice in any analysis; and (iii) that exploratory projects imply poor statistical practice. We show how our formal approach can inform and shape debates about such methodological claims.

A decade of theory as reflected in Psychological Science (2009-2019)

The dominant belief is that science progresses by testing theories and moving towards theoretical consensus. While it’s implicitly assumed that psychology operates in this manner, critical discussions claim that the field suffers from a lack of cumulative theory. To examine this paradox, we analysed research published in Psychological Science from 2009–2019 (N = 2,225). We found mention of 359 theories in-text, most were referred to only once. Only 53.66% of all manuscripts included the word theory, and only 15.33% explicitly claimed to test predictions derived from theories. We interpret this to suggest that the majority of research published in this flagship journal is not driven by theory, nor can it be contributing to cumulative theory building. These data provide insight into the kinds of research psychologists are conducting and raises questions about the role of theory in the psychological sciences.