Scientific Utopia III: Crowdsourcing Science

A scoping review on what constitutes a good research culture

Background

The crisis in research culture is well documented, covering issues such as a tendency for quantity over quality, unhealthy competitive environments, and assessment based on publications, journal prestige and funding. In response, research institutions need to assess their own practices to promote and advocate for change in the current research ecosystem. The purpose of the scoping review was to explore ' What does the evidence say about the 'problem' with 'poor' research culture, what are the benefits of 'good' research culture, and what does 'good' look like?'

Aims

To examine the peer-reviewed and grey literature to explore the interplay between research culture, open research, career paths, recognition and rewards, and equality, diversity, and inclusion, as part of a larger programme of activity for a research institution.

Methods

A scoping review was undertaken. Six databases were searched along with grey literature. Eligible literature had relevance to academic research institutions, addressed research culture, and were published between January 2017 to May 2022. Evidence was mapped and themed to specific categories. The search strategy, screening and analysis took place between April-May 2022.

Results

1666 titles and abstracts, and 924 full text articles were assessed for eligibility. Of these, 253 articles met the eligibility criteria for inclusion. A purposive sampling of relevant websites was drawn from to complement the review, resulting in 102 records included in the review. Key areas for consideration were identified across the four themes of job security, wellbeing and equality of opportunity, teamwork and interdisciplinary, and research quality and accountability.

Conclusions

There are opportunities for research institutions to improve their own practice, however institutional solutions cannot act in isolation. Research institutions and research funders need to work together to build a more sustainable and inclusive research culture that is diverse in nature and supports individuals' well-being, career progression and performance.

Addressing climate change with behavioral science: A global intervention tournament in 63 countries

Effectively reducing climate change requires marked, global behavior change. However, it is unclear which strategies are most likely to motivate people to change their climate beliefs and behaviors. Here, we tested 11 expert-crowdsourced interventions on four climate mitigation outcomes: beliefs, policy support, information sharing intention, and an effortful tree-planting behavioral task. Across 59,440 participants from 63 countries, the interventions' effectiveness was small, largely limited to nonclimate skeptics, and differed across outcomes: Beliefs were strengthened mostly by decreasing psychological distance (by 2.3%), policy support by writing a letter to a future-generation member (2.6%), information sharing by negative emotion induction (12.1%), and no intervention increased the more effortful behavior-several interventions even reduced tree planting. Last, the effects of each intervention differed depending on people's initial climate beliefs. These findings suggest that the impact of behavioral climate interventions varies across audiences and target behaviors.

Big team science initiatives: A catalyst for trustworthy advancements in IO psychology

Keener et al. (2023) raise concerns about the trustworthiness of Industrial/Organizational (IO) Psychology research and related fields due to the low reproducibility and replicability of research findings. The authors provide various solutions to resolve this crisis, such as improving training, realigning incentives, and adopting open science practices. Our commentary elaborates on one solution to which they briefly allude: Big Team Science Initiatives (BTSIs). BTSIs allow scholars to address the trustworthiness of our science by facilitating large sample theory testing, sharing and allocating resources, and selecting appropriate research strategies, all of which support the reproducibility and replication of research. Further, we propose that BTSIs may facilitate researcher training, encourage data sharing and materials, and realign incentives in our field. We discuss how BTSIs could be implemented in IO psychology and related fields, identifying and drawing upon similar BTSIs in related disciplines. Thus, our commentary is an extension of the focal article, encouraging scholars to collaboratively address the "crisis of confidence" facing our field using a big team science approach.

A Metascientific Review of the Evidential Value of Acceptance and Commitment Therapy for Depression

In the past three-and-a-half decades, nearly 500 randomized controlled trials (RCTs) have examined Acceptance and Commitment Therapy (ACT) for a range of health problems, including depression. However, emerging concerns regarding the replicability of scientific findings across psychology and mental health treatment outcome research highlight a need to re-examine the strength of evidence for treatment efficacy. Therefore, we conducted a metascientific review of the evidential value of ACT in treating depression. Whereas reporting accuracy was generally high across all trials, we found important differences in evidential value metrics corresponding to the types of control conditions used. RCTs of ACT compared to weaker controls (e.g., no treatment, waitlist) were well-powered, with sample sizes appropriate for detecting plausible effect sizes. They typically yielded stronger Bayesian evidence for (and larger posterior estimates of) ACT efficacy, though there was some evidence of significance inflation among these effects. RCTs of ACT against stronger controls (e.g., other psychotherapies), meanwhile, were poorly powered, designed to detect implausibly large effect sizes, and yielded ambiguous-if not contradicting-Bayesian evidence and estimates of efficacy. Although our review supports a view of ACT as efficacious for treating depression compared to weaker controls, future RCTs must provide more transparent reporting with larger groups of participants to properly assess the difference between ACT and competitor treatments such as behavioral activation and other forms of cognitive behavioral therapy. Clinicians and health organizations should reassess the use of ACT for depression if costs and resources are higher than for other efficacious treatments. Clinical trials contributing effects to our synthesis can be found at https://osf.io/qky35.

The trustworthiness of the cumulative knowledge in industrial/organizational psychology: The current state of affairs and a path forward

The goal of industrial/organizational (IO) psychology, is to build and organize trustworthy knowledge about people-related phenomena in the workplace. Unfortunately, as with other scientific disciplines, our discipline may be experiencing a "crisis of confidence" stemming from the lack of reproducibility and replicability of many of our field's research findings, which would suggest that much of our research may be untrustworthy. If a scientific discipline's research is deemed untrustworthy, it can have dire consequences, including the withdraw of funding for future research. In this focal article, we review the current state of reproducibility and replicability in IO psychology and related fields. As part of this review, we discuss factors that make it less likely that research findings will be trustworthy, including the prevalence of scientific misconduct, questionable research practices (QRPs), and errors. We then identify some root causes of these issues and provide several potential remedies. In particular, we highlight the need for improved research methods and statistics training as well as a re-alignment of the incentive structure in academia. To accomplish this, we advocate for changes in the reward structure, improvements to the peer review process, and the implementation of open science practices. Overall, addressing the current "crisis of confidence" in IO psychology requires individual researchers, academic institutions, and publishers to embrace system-wide change.

CIAO: a living experiment in interdisciplinary large-scale collaboration facilitated by the Adverse Outcome Pathway framework

Introduction

The CIAO project was launched in Spring 2020 to address the need to make sense of the numerous and disparate data available on COVID-19 pathogenesis. Based on a crowdsourcing model of large-scale collaboration, the project has exploited the Adverse Outcome Pathway (AOP) knowledge management framework built to support chemical risk assessment driven by mechanistic understanding of the biological perturbations at the different organizational levels. Hence the AOPs might have real potential to integrate data produced through different approaches and from different disciplines as experienced in the context of COVID-19. In this study, we aim to address the effectiveness of the AOP framework (i) in supporting an interdisciplinary collaboration for a viral disease and (ii) in working as the conceptual mediator of a crowdsourcing model of collaboration.

Methods

We used a survey disseminated among the CIAO participants, a workshop open to all interested CIAO contributors, a series of interviews with some participants and a self-reflection on the processes.

Results

The project has supported genuine interdisciplinarity with exchange of knowledge. The framework provided a common reference point for discussion and collaboration. The diagram used in the AOPs assisted with making explicit what are the different perspectives brought to the knowledge about the pathways. The AOP-Wiki showed up many aspects about its usability for those not already in the world of AOPs. Meanwhile their use in CIAO highlighted needed adaptations. Introduction of new Wiki elements for modulating factors was potentially the most disruptive one. Regarding how well AOPs support a crowdsourcing model of large-scale collaboration, the CIAO project showed that this is successful when there is a strong central organizational impetus and when clarity about the terms of the collaboration is brought as early as possible.

Discussion

Extrapolate the successful CIAO approach and related processes to other areas of science where the AOP could foster interdisciplinary and systematic organization of the knowledge is an exciting perspective.

Dynamic interlinkages between the crude oil and gold and stock during Russia-Ukraine War: evidence from an extended TVP-VAR analysis

The Russia-Ukraine significantly influences the oil market. We employ a time-varying parameter vector autoregression (TVP-VAR) in combination with an extended joint connectedness approach to identify the sources of the oil market's volatility by studying interlinkages between the crude oil and gold and stock market by characterizing the connectedness of four markets starting from January 1, 2018 to April 8, 2022. Our attention is mostly paid to the period marked by the event that Russia invaded Ukraine on 24 February, 2022. Our results demonstrate that the war shocks appear to influence the system-wide dynamic connectedness, which signifies the interlinkages among the considered markets. Net total directional connectedness suggests that the oil and gold markets appear to be the net transmitter of spillover shocks in the system. However, there are shifts in the roles of these two markets during the time of the Russia-Ukraine war shock. Pairwise connectedness highlights the significance of the oil market in transmitting the adverse influences of shocks to other markets, especially during the Russia-Ukraine war.

Storm after the Gloomy days: Influences of COVID-19 pandemic on volatility of the energy market

Volatility is a common phenomenon in the energy market, but COVID-19 has cast a dark shadow over this characteristic. In light of this observation, individuals might have an incorrect impression of the impact of this shock on the energy markets. By applying a time-varying parameter vector autoregression (TVP-VAR) in combination with an extended joint connectedness approach to identify the sources of the energy market's volatility, we characterize the influences of COVID-19 health crisis and the volatility of the crude oil and precious metals (including gold and silver) market on the volatility of the energy market starting from January 1, 2020, to December 31, 2021. The total connectedness index, the net total, and pairwise directional connectedness measures obtained from the extended TVP-VAR allow us to monitor interlinkages from various variables in a designed network. The novel method has the benefit of distinguishing between a net recipient and a net transmitter. Our results demonstrate that the COVID-19 pandemic shocks first absorb the volatility from the energy and precious market to cause lagged but more severe consequences returning to these markets. Furthermore, there is a time-variant of system-wide interlinkages. Net total directional connectedness suggests that the oil and gold markets consistently appear to be a net transmitter of spillover shocks in the energy market. The COVID-19 pandemic shock first plays the role of shock receiver from other markets. However, this uncertainty shock acts as a shock transmitter, and its effects seem to be delayed but persistent for an extended period, thus making the energy and precious metal markets more volatile.

Understanding crowdsourcing in science

Over the past 16 years, the concept of crowdsourcing has rapidly gained traction across many research fields. While related debates focused mainly on its importance for business, the public and non-governmental sectors, its relevance for generating scientific knowledge is increasingly emphasized. This rising interest remains in contradiction with its feeble recognition, and excessive simplifications reducing crowdsourcing in science to citizen science. Conceptual clarity and a coherent framework would help integrate the various research streams. The aim of this paper is to extend reflection on crowdsourcing in science by analyzing the characteristics of the phenomenon. We synthesize a consensual definition from the literature, and structure key characteristics into a coherent framework, useful in guiding further research. We use a systematic literature review procedure to generate a pool of 42 definitions from a comprehensive set of 62 articles spanning different literatures, including: business and economics, education, psychology, biology, and communication studies. We follow a mixed-method approach that combines bibliometric and frequency analyses with deductive coding and thematic analysis. Based on triangulated results we develop an integrative definition: crowdsourcing in science is a collaborative online process through which scientists involve a group of self-selected individuals of varying, diverse knowledge and skills, via an open call to the Internet and/or online platforms, to undertake a specified research task or set of tasks. We also provide a conceptual framework that covers four key characteristics: initiator, crowd, process, and technology.

Peer review in funding-by-lottery: A systematic overview and expansion

Despite the surging interest in introducing lottery mechanisms into decision-making procedures for science funding bodies, the discourse on funding-by-lottery remains underdeveloped and, at times, misleading. Funding-by-lottery is sometimes presented as if it were a single mechanism when, in reality, there are many funding-by-lottery mechanisms with important distinguishing features. Moreover, funding-by-lottery is sometimes portrayed as an alternative to traditional methods of peer review when peer review is still used within funding-by-lottery approaches. This obscures a proper analysis of the (hypothetical and actual) variants of funding-by-lottery and important differences amongst them. The goal of this article is to provide a preliminary taxonomy of funding-by-lottery variants and evaluate how the existing evidence on peer review might lend differentiated support for variants of funding-by-lottery. Moreover, I point to gaps in the literature on peer review that must be addressed in future research. I conclude by building off of the work of Avin in moving toward a more holistic evaluation of funding-by-lottery. Specifically, I consider implications funding-by-lottery variants may have regarding trust and social responsibility.

The Benefits, Barriers, and Risks of Big-Team Science

Progress in psychology has been frustrated by challenges concerning replicability, generalizability, strategy selection, inferential reproducibility, and computational reproducibility. Although often discussed separately, these five challenges may share a common cause: insufficient investment of intellectual and nonintellectual resources into the typical psychology study. We suggest that the emerging emphasis on big-team science can help address these challenges by allowing researchers to pool their resources together to increase the amount available for a single study. However, the current incentives, infrastructure, and institutions in academic science have all developed under the assumption that science is conducted by solo principal investigators and their dependent trainees, an assumption that creates barriers to sustainable big-team science. We also anticipate that big-team science carries unique risks, such as the potential for big-team-science organizations to be co-opted by unaccountable leaders, become overly conservative, and make mistakes at a grand scale. Big-team-science organizations must also acquire personnel who are properly compensated and have clear roles. Not doing so raises risks related to mismanagement and a lack of financial sustainability. If researchers can manage its unique barriers and risks, big-team science has the potential to spur great progress in psychology and beyond.

Towards a typology development of crowdsourcing in science

Crowdsourcing in science as collaborative online process through which non-professional and/or professional scientists incorporate a group of individuals of varying, diversity knowledge and skills, via an open call to the Internet and/or online platforms, to undertaking of a task in science, is an important strategy to support scientific research that has gained attention in academia and practitioners. While research efforts to date have focused on the benefits of crowdsourcing in science, its typology has yet to mature. Typologies are important in describing complex, multidisciplinary organisational forms such as crowdsourcing in science. The main purpose of this article is to identify and provide a typology of crowdsourcing in science. Based on the thematic analysis of publications collected in a systematic manner and focused group interviews, 12 types of crowdsourcing in science are identified. The proposed crowdsourcing in science typology matrix may be a starting point for future research and decision-making by practitioners regarding the choice of a specific type of crowdsourcing in science.

Accelerating Psychological Science With Metastudies: A Demonstration Using the Risky-Choice Framing Effect

A metastudy is a set of many tiny studies (microstudies) created from a much larger collection of possibilities. Metastudies can yield many of the benefits of time-consuming replications and meta-analyses but more efficiently and with greater attention to generalizability and the causal effects of moderators. Statistical precision and power are higher than in studies with the same total sample size but with fewer conditions and more participants per condition. In this article, we describe metastudies and their benefits, demonstrate how to conduct a metastudy using the well-known risky-choice framing effect as an example, and report on the generalizability of that effect. In three metastudies (total N = 2,338), the framing effect generalized well across most of the potential moderators tested, as was expected. Surprisingly, however, the effect was up to twice as large when the certain option was replaced with a slightly risky option; prospect theory predicts the opposite, and fuzzy-trace theory predicts no difference. Metastudies provide a relatively quick and not-so-painful way of examining an effect's generalizability without waiting for a meta-analysis. Both individual labs and multilab networks are encouraged to shift from traditional studies to metastudies.

Reproducibility in the Social Sciences

Concern over social scientists' inability to reproduce empirical research has spawned a vast and rapidly growing literature. The size and growth of this literature make it difficult for newly interested academics to come up to speed. Here, we provide a formal text modeling approach to characterize the entirety of the field, which allows us to summarize the breadth of this literature and identify core themes. We construct and analyze text networks built from 1,947 articles to reveal differences across social science disciplines within the body of reproducibility publications and to discuss the diversity of subtopics addressed in the literature. This field-wide view suggests that reproducibility is a heterogeneous problem with multiple sources for errors and strategies for solutions, a finding that is somewhat at odds with calls for largely passive remedies reliant on open science. We propose an alternative rigor and reproducibility model that takes an active approach to rigor prior to publication, which may overcome some of the shortfalls of the postpublication model.

Intervention Tournaments: An Overview of Concept, Design, and Implementation

A large portion of research in the social sciences is devoted to using interventions to combat societal and social problems, such as prejudice, discrimination, and intergroup conflict. However, these interventions are often developed using the theories and/or intuitions of the individuals who developed them and evaluated in isolation without comparing their efficacy with other interventions. Here, we make the case for an experimental design that addresses such issues: an intervention tournament-that is, a study that compares several different interventions against a single control and uses the same standardized outcome measures during assessment and participants drawn from the same population. We begin by highlighting the utility of intervention tournaments as an approach that complements other, more commonly used approaches to addressing societal issues. We then describe various approaches to intervention tournaments, which include crowdsourced, curated, and in-house-developed intervention tournaments, and their unique characteristics. Finally, we discuss practical recommendations and key design insights for conducting such research, given the existing literature. These include considerations of intervention-tournament deployment, characteristics of included interventions, statistical analysis and reporting, study design, longitudinal and underlying psychological mechanism assessment, and theoretical ramifications.

KAZNET: An Open-Source, Micro-Tasking Platform for Remote Locations

Field surveys are the workhorse of social and environmental research, but conventional collection through monitors or enumerators are cost prohibitive in many remote or otherwise difficult settings, which can lead to a poor understanding of those environments and an underrepresentation of the people living in them. In such cases, micro-tasking can offer a promising alternative. By activating in-situ data collectors, micro-tasking avoids many of the large expenses related to conventional field survey processes. In addition to relaxing resource constraints, crowd-sourcing can be flexible and employ data quality protocols unheard-of for conventional methods. This study assesses the potential of using micro-tasking to monitor socioeconomic and environmental indicators in remote settings using a new platform called KAZNET. KAZNET leverages the network of people with smartphones, which are becoming ubiquitous even in the remote rural settings, to execute both long-term and short-term data collection activities, with flexibility to adjust or add tasks in real-time. It also allows for multiple projects, requiring different data types, to be rolled out in the same platform simultaneously. For the data-collector, KAZNET is effectively a wrapper for the commonly used and open source, Open Data Kit (ODK) software, which specializes in offline data collection. A web interface allows administrators to calibrate, deploy, and validate tasks performed by contributors. KAZNET has been used in several projects to collect data in remote pastoral regions of East Africa since its inception in 2017. KAZNET has shown to be effective for collecting high frequency and repeated measures from markets, households and rangelands in remote regions at relatively low cost compared to traditional survey methods. While the successes of micro-tasking are promising, there are clear trade-offs and complementarities between micro-tasking and standard surveys methods, which researchers and practitioners need to consider when implementing either approach.

Towards a culture of open scholarship: the role of pedagogical communities

The UK House of Commons Science and Technology Committee has called for evidence on the roles that different stakeholders play in reproducibility and research integrity. Of central priority are proposals for improving research integrity and quality, as well as guidance and support for researchers. In response to this, we argue that there is one important component of research integrity that is often absent from discussion: the pedagogical consequences of how we teach, mentor, and supervise students through open scholarship. We justify the need to integrate open scholarship principles into research training within higher education and argue that pedagogical communities play a key role in fostering an inclusive culture of open scholarship. We illustrate these benefits by presenting the Framework for Open and Reproducible Research Training (FORRT), an international grassroots community whose goal is to provide support, resources, visibility, and advocacy for the adoption of principled, open teaching and mentoring practices, whilst generating conversations about the ethics and social impact of higher-education pedagogy. Representing a diverse group of early-career researchers and students across specialisms, we advocate for greater recognition of and support for pedagogical communities, and encourage all research stakeholders to engage with these communities to enable long-term, sustainable change.

Citizen Science for Traffic Monitoring: Investigating the Potentials for Complementing Traffic Counters with Crowdsourced Data

Traffic counts are among the most frequently employed data to assess the traffic patterns and key performance indicators of next generation sustainable cities. Automatised counting is often based on conventional traffic monitoring systems such as inductive loop counters (ILCs). These are costly to install, maintain, and support. In this paper, we investigate the possibilities to complement and potentially replace the existing traffic monitoring infrastructure with crowdsourcing solutions. More precisely, we investigate the capabilities to predict the ILC-obtained data using Telraam counters, low-cost camera counters voluntarily employed by citizens and freely accessible by the general public. In this context, we apply different exploratory data analysis approaches and demonstrate a regression procedure with a selected set of regression models. The presented analysis is demonstrated on different urban and highway road segments in Slovenia. Our results show that the data obtained from low-cost and easily accessible counters can be used to replace the existing traffic monitoring infrastructure in different scenarios. These results confirm the prospective to directly apply the citizen engagement in the process of planning and maintaining sustainable future cities.

Crowdsourcing Global Perspectives in Ecology Using Social Media

Transparent, open, and reproducible research is still far from routine, and the full potential of open science has not yet been realized. Crowdsourcing–defined as the usage of a flexible open call to a heterogeneous group of individuals to recruit volunteers for a task –is an emerging scientific model that encourages larger and more outwardly transparent collaborations. While crowdsourcing, particularly through citizen- or community-based science, has been increasing over the last decade in ecological research, it remains infrequently used as a means of generating scientific knowledge in comparison to more traditional approaches. We explored a new implementation of crowdsourcing by using an open call on social media to assess its utility to address fundamental ecological questions. We specifically focused on pervasive challenges in predicting, mitigating, and understanding the consequences of disturbances. In this paper, we briefly review open science concepts and their benefits, and then focus on the new methods we used to generate a scientific publication. We share our approach, lessons learned, and potential pathways forward for expanding open science. Our model is based on the beliefs that social media can be a powerful tool for idea generation and that open collaborative writing processes can enhance scientific outcomes. We structured the project in five phases: (1) draft idea generation, (2) leadership team recruitment and project development, (3) open collaborator recruitment via social media, (4) iterative paper development, and (5) final editing, authorship assignment, and submission by the leadership team. We observed benefits including: facilitating connections between unusual networks of scientists, providing opportunities for early career and underrepresented groups of scientists, and rapid knowledge exchange that generated multidisciplinary ideas. We also identified areas for improvement, highlighting biases in the individuals that self-selected participation and acknowledging remaining barriers to contributing new or incompletely formed ideas into a public document. While shifting scientific paradigms to completely open science is a long-term process, our hope in publishing this work is to encourage others to build upon and improve our efforts in new and creative ways.

Small Effects: The Indispensable Foundation for a Cumulative Psychological Science

We draw on genetics research to argue that complex psychological phenomena are most likely determined by a multitude of causes and that any individual cause is likely to have only a small effect. Building on this, we highlight the dangers of a publication culture that continues to demand large effects. First, it rewards inflated effects that are unlikely to be real and encourages practices likely to yield such effects. Second, it overlooks the small effects that are most likely to be real, hindering attempts to identify and understand the actual determinants of complex psychological phenomena. We then explain the theoretical and practical relevance of small effects, which can have substantial consequences, especially when considered at scale and over time. Finally, we suggest ways in which scholars can harness these insights to advance research and practices in psychology (i.e., leveraging the power of big data, machine learning, and crowdsourcing science; promoting rigorous preregistration, including prespecifying the smallest effect size of interest; contextualizing effects; changing cultural norms to reward accurate and meaningful effects rather than exaggerated and unreliable effects). Only once small effects are accepted as the norm, rather than the exception, can a reliable and reproducible cumulative psychological science be built.

Impacts of COVID-19 pandemic on the global energy system and the shift progress to renewable energy: Opportunities, challenges, and policy implications

Being declared a global emergency, the COVID-19 pandemic has taken many lives, threatened livelihoods and businesses around the world. The energy industry, in particular, has experienced tremendous pressure resulting from the pandemic. In response to such a challenge, the development of sustainable resources and renewable energy infrastructure has demonstrated its potential as a promising and effective strategy. To sufficiently address the effect of COVID-19 on renewable energy development strategies, short-term policy priorities should be identified, while mid-term and long-term action plans should be formulated in achieving the well-defined renewable energy targets and progress towards a more sustainable energy future. In this review, opportunities, challenges, and significant impacts of the COVID-19 pandemic on current and future sustainable energy strategies were analyzed in detail; while drawing from experiences in identifying reasonable behaviors, orientating appropriate actions, and policy implications on the sustainable energy trajectory were also mentioned. Indeed, the question is that whether the COVID-19 pandemic will kill us or provide us with a precious lesson on future sustainable energy development.

Using prediction markets to predict the outcomes in the Defense Advanced Research Projects Agency's next-generation social science programme

There is evidence that prediction markets are useful tools to aggregate information on researchers' beliefs about scientific results including the outcome of replications. In this study, we use prediction markets to forecast the results of novel experimental designs that test established theories. We set up prediction markets for hypotheses tested in the Defense Advanced Research Projects Agency's (DARPA) Next Generation Social Science (NGS2) programme. Researchers were invited to bet on whether 22 hypotheses would be supported or not. We define support as a test result in the same direction as hypothesized, with a Bayes factor of at least 10 (i.e. a likelihood of the observed data being consistent with the tested hypothesis that is at least 10 times greater compared with the null hypothesis). In addition to betting on this binary outcome, we asked participants to bet on the expected effect size (in Cohen's d) for each hypothesis. Our goal was to recruit at least 50 participants that signed up to participate in these markets. While this was the case, only 39 participants ended up actually trading. Participants also completed a survey on both the binary result and the effect size. We find that neither prediction markets nor surveys performed well in predicting outcomes for NGS2.

Sameness and Difference in Psychological Research on Consensually Non-Monogamous Relationships: The Need for Invariance and Equivalence Testing

Comparative research involving consensually non-monogamous (CNM) relationships and outcomes related to well-being continues to grow as an area of interest within sexual science. However, claims of sameness and/or difference between groups rely on two critical, yet widely under-appreciated assumptions: that the concepts being compared between groups are the same (i.e., measurement invariance), and that logically and statistically coherent procedures are used for evaluating sameness (i.e., equivalence testing). We evaluated the state of measurement invariance and equivalence across three studies, involving different types of CNM comparisons (i.e., relationship types, partner types) and designs (analysis of primary individual data, primary dyadic data, and secondary data). Our invariance tests of CNM compared to monogamous individuals (Study 1) and "primary" compared to "secondary" partners in dyadic appraisal of CNM individuals (Study 2) revealed that many measures of well-being failed to replicate their measurement models and were not generalizable across relationship types or partner types. Our reanalyses of existing comparative CNM effects using individual and meta-analyzed equivalence tests (Study 3), meanwhile, indicated that this literature requires more consistent reporting practices and larger samples, as most studies produced uninformative tests of equivalence. Our results illustrate the importance of auxiliary hypothesis evaluation and statistical procedure selection for generating informative comparative tests. Our findings also highlight potential divergences in social construction of well-being. We offer suggestions for researchers, reviewers, and editors in terms of needed methodological reforms for future comparative CNM research.

Les pratiques de recherche ouvertes en psychologie

Cet article vise à offrir une vision d’ensemble des récentes évolutions des pratiques de recherche en psychologie. Un rappel des différents symptômes de la crise de la réplicabilité (et de confiance) ayant affecté la psychologie sera suivi par une discussion approfondie et nuancée des facteurs responsables de cette situation. Il s’agira ensuite, en s’appuyant sur des illustrations et des ressources, de démontrer le rôle crucial des pratiques de recherche ouvertes comme moyen de résoudre ces difficultés. La connaissance et l’adoption de ces pratiques de recherche popularisées par le mouvement de la science ouverte sont indispensables afin de contribuer, via la transparence et l’ouverture, à l’effort collectif d’amélioration de la fiabilité et de la réplicabilité des résultats en psychologie.

Heterogeneity of Research Results: A New Perspective From Which to Assess and Promote Progress in Psychological Science

Heterogeneity emerges when multiple close or conceptual replications on the same subject produce results that vary more than expected from the sampling error. Here we argue that unexplained heterogeneity reflects a lack of coherence between the concepts applied and data observed and therefore a lack of understanding of the subject matter. Typical levels of heterogeneity thus offer a useful but neglected perspective on the levels of understanding achieved in psychological science. Focusing on continuous outcome variables, we surveyed heterogeneity in 150 meta-analyses from cognitive, organizational, and social psychology and 57 multiple close replications. Heterogeneity proved to be very high in meta-analyses, with powerful moderators being conspicuously absent. Population effects in the average meta-analysis vary from small to very large for reasons that are typically not understood. In contrast, heterogeneity was moderate in close replications. A newly identified relationship between heterogeneity and effect size allowed us to make predictions about expected heterogeneity levels. We discuss important implications for the formulation and evaluation of theories in psychology. On the basis of insights from the history and philosophy of science, we argue that the reduction of heterogeneity is important for progress in psychology and its practical applications, and we suggest changes to our collective research practice toward this end.

Optimizing the scientific study of suicide with open and transparent research practices

Suicide research is vitally important, yet-like psychology research more broadly-faces methodological challenges. In recent years, researchers have raised concerns about standard practices in psychological research, concerns that apply to suicide research and raise questions about its robustness and validity. In the present paper, we review these concerns and the corresponding solutions put forth by the "open science" community. These include using open science platforms, pre-registering studies, ensuring reproducible analyses, using high-powered studies, ensuring open access to research materials and products, and conducting replication studies. We build upon existing guides, address specific obstacles faced by suicide researchers, and offer a clear set of recommended practices for suicide researchers. In particular, we consider challenges that suicide researchers may face in seeking to adopt "open science" practices (e.g., prioritizing large samples) and suggest possible strategies that the field may use in order to ensure robust and transparent research, despite these challenges.

Gratitude Questionnaire-20 Items (G20): A Cross-Cultural, Psychometric and Crowdsourcing Analysis

The use in psychology of crowdsourcing platforms as a method of data collection has been increasing in popularity because of its relative ease and versatility. Our goal is to adapt the Gratitude Questionnaire–20 Items (G20) to the English language by using data collected through a crowdsourcing platform. The G20 is a comprehensive instrument that takes in consideration the different basic processes of gratitude and assesses the construct’s cognitive, evaluative, emotional, and behavioral processes. We test the psychometric properties of the English version of the G20 with a Prolific (ProA) user sample. We assess the adequacy of the G20 for the crowdsourcing population in its English version. A description of the characteristics of the participants is conducted. Reliability analyses reveal an optimal internal consistency of the adapted scale. The results are discussed from a cross-cultural vision of gratitude. We conclude that the Gratitude Questionnaire–20 Items (G20), adapted to English with an American sample, is a psychometrically strong instrument to measure gratitude using crowdsourcing platforms for data collection and, therefore, a reference and useful tool in future research.

Using Community Science to Reveal the Global Chemogeography of River Metabolomes

River corridor metabolomes reflect organic matter (OM) processing that drives aquatic biogeochemical cycles. Recent work highlights the power of ultrahigh-resolution mass spectrometry for understanding metabolome composition and river corridor metabolism. However, there have been no studies on the global chemogeography of surface water and sediment metabolomes using ultrahigh-resolution techniques. Here, we describe a community science effort from the Worldwide Hydrobiogeochemistry Observation Network for Dynamic River Systems (WHONDRS) consortium to characterize global metabolomes in surface water and sediment that span multiple stream orders and biomes. We describe the distribution of key aspects of metabolomes including elemental groups, chemical classes, indices, and inferred biochemical transformations. We show that metabolomes significantly differ across surface water and sediment and that surface water metabolomes are more rich and variable. We also use inferred biochemical transformations to identify core metabolic processes shared among surface water and sediment. Finally, we observe significant spatial variation in sediment metabolites between rivers in the eastern and western portions of the contiguous United States. Our work not only provides a basis for understanding global patterns in river corridor biogeochemical cycles but also demonstrates that community science endeavors can enable global research projects that are unfeasible with traditional research models.

Changing the logic of replication: A case from infant studies

Among infant researchers there is growing concern regarding the widespread practice of undertaking studies that have small sample sizes and employ tests with low statistical power (to detect a wide range of possible effects). For many researchers, issues of confidence may be partially resolved by relying on replications. Here, we bring further evidence that the classical logic of confirmation, according to which the result of a replication study confirms the original finding when it reaches statistical significance, could be usefully abandoned. With real examples taken from the infant literature and Monte Carlo simulations, we show that a very wide range of possible replication results would in a formal statistical sense constitute confirmation as they can be explained simply due to sampling error. Thus, often no useful conclusion can be derived from a single or small number of replication studies. We suggest that, in order to accumulate and generate new knowledge, the dichotomous view of replication as confirmatory/disconfirmatory can be replaced by an approach that emphasizes the estimation of effect sizes via meta-analysis. Moreover, we discuss possible solutions for reducing problems affecting the validity of conclusions drawn from meta-analyses in infant research.

Validity of energy social research during and after COVID-19: challenges, considerations, and responses

Measures to control the spread of coronavirus disease 2019 (COVID-19) are having unprecedented impacts on people's lives around the world. In this paper, we argue that those conducting social research in the energy domain should give special consideration to the internal and external validity of their work conducted during this pandemic period. We set out a number of principles that researchers can consider to give themselves and research users greater confidence that findings and recommendations will still be applicable in years to come. Largely grounded in existing good practice guidance, our recommendations include collecting and reporting additional supporting contextual data, reviewing aspects of research design for vulnerability to validity challenges, and building in longitudinal elements where feasible. We suggest that these approaches also bring a number of opportunities to generate new insights. However, we caution that a more systemic challenge to validity of knowledge produced during this period may result from changes in the kinds of social research that it is practicable to pursue.