Data sharing in psychology

Open science practices in the false memory literature

In response to the replication crisis in psychology, the scientific community has advocated open science practices to promote transparency and reproducibility. Although existing reviews indicate inconsistent and generally low adoption of open science in psychology, a current-day, detailed analysis is lacking. Recognising the significant impact of false memory research in legal contexts, we conducted a preregistered systematic review to assess the integration of open science practices within this field, analysing 388 publications from 2015 to 2023 (including 15 replications and 3 meta-analyses). Our findings indicated a significant yet varied adoption of open science practices. Most studies (86.86%) adhered to at least one measure, with publication accessibility being the most consistently adopted practice at 73.97%. While data sharing demonstrated the most substantial growth, reaching about 75% by 2023, preregistration and analysis script sharing lagged, with 20-25% adoption in 2023. This review highlights a promising trend towards enhanced research quality, transparency, and reproducibility in false memory research. However, the inconsistent implementation of open science practices may still challenge the verification, replication, and interpretation of research findings. Our study underscores the need for a comprehensive adoption of open science to improve research reliability and validity substantially, fostering trust and credibility in psychology.

Data-Sharing Across Otolaryngology: Comparing Journal Policies and Their Adherence to the FAIR Principles

OBJECTIVE

Data-sharing plays an essential role in advancing scientific understanding. Here, we aim to identify the commonalities and differences in data-sharing policies endorsed by otolaryngology journals and to assess their adherence to the FAIR (findable, accessible, interoperable, reusable) principles.

METHODS

Data-sharing policies were searched for among 111 otolaryngology journals, as listed by Scimago Journal & Country Rank. Policy extraction of the top biomedical journals as ranked by Google Scholar metrics were used as a comparison. The FAIR principles for scientific data management and stewardship were used for the extraction framework. This occurred in a blind, masked, and independent fashion.

RESULTS

Of the 111 ranked otolaryngology journals, 100 met inclusion criteria. Of those 100 journals, 79 provided data-sharing policies. There was a clear lack of standardization across policies, along with specific gaps in accessibility and reusability which need to be addressed. Seventy-two policies (of 79; 91%) designated that metadata should have globally unique and persistent identifiers. Seventy-one (of 79; 90%) policies specified that metadata should clearly include the identifier of the data they describe. Fifty-six policies (of 79; 71%) outlined that metadata should be richly described with a plurality of accurate and relevant attributes.

CONCLUSION

Otolaryngology journals have varying data-sharing policies, and adherence to the FAIR principles appears to be moderate. This calls for increased data transparency, allowing for results to be reproduced, confirmed, and debated.

When open data closes the door: A critical examination of the past, present and the potential future for open data guidelines in journals

Opening data promises to improve research rigour and democratize knowledge production. But it also presents practical, theoretical, and ethical considerations for qualitative researchers in particular. Discussion about open data in qualitative social psychology predates the replication crisis. However, the nuances of this ongoing discussion have not been translated into current journal guidelines on open data. In this article, we summarize ongoing debates about open data from qualitative perspectives, and through a content analysis of 261 journals we establish the state of current journal policies for open data in the domain of social psychology. We critically discuss how current common expectations for open data may not be adequate for establishing qualitative rigour, can introduce ethical challenges, and may place those who wish to use qualitative approaches at a disadvantage in peer review and publication processes. We advise that future open data guidelines should aim to reflect the nuance of arguments surrounding data sharing in qualitative research, and move away from a universal "one-size-fits-all" approach to data sharing. This article outlines the past, present, and the potential future of open data guidelines in social-psychological journals. We conclude by offering recommendations for how journals might more inclusively consider the use of open data in qualitative methods, whilst recognizing and allowing space for the diverse perspectives, needs, and contexts of all forms of social-psychological research.

Reasons for qualitative psychologists to share human data

Qualitative data sharing practices in psychology have not developed as rapidly as those in parallel quantitative domains. This is often explained by numerous epistemological, ethical and pragmatic issues concerning qualitative data types. In this article, I provide an alternative to the frequently expressed, often reasonable, concerns regarding the sharing of qualitative human data by highlighting three advantages of qualitative data sharing. I argue that sharing qualitative human data is not by default 'less ethical', 'riskier' and 'impractical' compared with quantitative data sharing, but in some cases more ethical, less risky and easier to manage for sharing because (1) informed consent can be discussed, negotiated and validated; (2) the shared data can be curated by special means; and (3) the privacy risks are mainly local instead of global. I hope this alternative perspective further encourages qualitative psychologists to share their data when it is epistemologically, ethically and pragmatically possible.

Open Science Practices in Communication Sciences and Disorders: A Survey

PURPOSE

Open science is a collection of practices that seek to improve the accessibility, transparency, and replicability of science. Although these practices have garnered interest in related fields, it remains unclear whether open science practices have been adopted in the field of communication sciences and disorders (CSD). This study aimed to survey the knowledge, implementation, and perceived benefits and barriers of open science practices in CSD.

METHOD

An online survey was disseminated to researchers in the United States actively engaged in CSD research. Four-core open science practices were examined: preregistration, self-archiving, gold open access, and open data. Data were analyzed using descriptive statistics and regression models.

RESULTS

Two hundred twenty-two participants met the inclusion criteria. Most participants were doctoral students (38%) or assistant professors (24%) at R1 institutions (58%). Participants reported low knowledge of preregistration and gold open access. There was, however, a high level of desire to learn more for all practices. Implementation of open science practices was also low, most notably for preregistration, gold open access, and open data (< 25%). Predictors of knowledge and participation, as well as perceived barriers to implementation, are discussed.

CONCLUSION

Although participation in open science appears low in the field of CSD, participants expressed a strong desire to learn more in order to engage in these practices in the future. Supplemental Material and Open Science Form: https://doi.org/10.23641/asha.21569040.

Is open science a double-edged sword?: data sharing and the changing citation pattern of Chinese economics articles

Data sharing is an important part of open science (OS), and more and more institutions and journals have been enforcing open data (OD) policies. OD is advocated to help increase academic influences and promote scientific discovery and development, but such a proposition has not been elaborated on well. This study explores the nuanced effects of the OD policies on the citation pattern of articles by using the case of Chinese economics journals. China Industrial Economics (CIE) is the first and only Chinese social science journal so far to adopt a compulsory OD policy, requiring all published articles to share original data and processing codes. We use the article-level data and difference-in-differences (DID) approach to compare the citation performance of articles published in CIE and 36 comparable journals. Firstly, we find that the OD policy quickly increased the number of citations, and each article on average received 0.25, 1.19, 0.86, and 0.44 more citations in the first four years after publication respectively. Furthermore, we also found that the citation benefit of the OD policy rapidly decreased over time, and even became negative in the fifth year after publication. In conclusion, this changing citation pattern suggests that an OD policy can be double edged sword, which can quickly increase citation performance but simultaneously accelerate the aging of articles.

Clinical trial data-sharing policies among journals, funding agencies, foundations, and other professional organizations: a scoping review

BACKGROUND AND OBJECTIVES

To identify the similarities and differences in data-sharing policies for clinical trial data that are endorsed by biomedical journals, funding agencies, and other professional organizations. Additionally, to determine the beliefs, and opinions regarding data-sharing policies for clinical trials discussed in articles published in biomedical journals.

METHODS

Two searches were conducted, a bibliographic search for published articles that present beliefs, opinions, similarities, and differences regarding policies governing the sharing of clinical trial data. The second search analyzed the gray literature (non-peer-reviewed publications) to identify important data-sharing policies in selected biomedical journals, foundations, funding agencies, and other professional organizations.

RESULTS

A total of 471 articles were included after database search and screening, with 45 from the bibliographic search and 426 from the gray literature search. A total of 424 data-sharing policies were included. Fourteen of the 45 published articles from the bibliographic search (31.1%) discussed only advantages specific to data-sharing policies, 27 (27/45; 60%) discussed both advantages and disadvantages, and 4 (4/45; 8.9%) discussed only disadvantages specific. A total of 216 journals (of 270; 80%) specified a data-sharing policy provided by the journal itself. One hundred industry data-sharing policies were included, and 32 (32%) referenced a data-sharing policy on their website. One hundred and thirty-six (42%) organizations (of 327) specified a data-sharing policy.

CONCLUSION

We found many similarities listed as advantages to data-sharing and fewer disadvantages were discussed within the literature. Additionally, we found a wide variety of commonalities and differences-such as the lack of standardization between policies, and inadequately addressed details regarding the accessibility of research data-that exist in data-sharing policies endorsed by biomedical journals, funding agencies, and other professional organizations. Our study may not include information on all data sharing policies and our data is limited to the entities' descriptions of each policy.

Establishing ground truth in the traumatic brain injury literature: if replication is the answer, then what are the questions?

The replication crisis poses important challenges to modern science. Central to this challenge is re-establishing ground truths or the most fundamental theories that serve as the bedrock to a scientific community. However, the goal to identify hypotheses with the greatest support is non-trivial given the unprecedented rate of scientific publishing. In this era of high-volume science, the goal of this study is to sample from one research community within clinical neuroscience (traumatic brain injury) and track major trends that have shaped this literature over the past 50 years. To do so, we first conduct a decade-wise (1980-2019) network analysis to examine the scientific communities that shape this literature. To establish the robustness of our findings, we utilized searches from separate search engines (Web of Science; Semantic Scholar). As a second goal, we sought to determine the most highly cited hypotheses influencing the literature in each decade. In a third goal, we then searched for any papers referring to 'replication' or efforts to reproduce findings within our >50 000 paper dataset. From this search, 550 papers were analysed to determine the frequency and nature of formal replication studies over time. Finally, to maximize transparency, we provide a detailed procedure for the creation and analysis of our dataset, including a discussion of each of our major decision points, to facilitate similar efforts in other areas of neuroscience. We found that the unparalleled rate of scientific publishing within the brain injury literature combined with the scarcity of clear hypotheses in individual publications is a challenge to both evaluating accepted findings and determining paths forward to accelerate science. Additionally, while the conversation about reproducibility has increased over the past decade, the rate of published replication studies continues to be a negligible proportion of the research. Meta-science and computational methods offer the critical opportunity to assess the state of the science and illuminate pathways forward, but ultimately there is structural change needed in the brain injury literature and perhaps others.

Enhancing Data Integration, Interoperability, and Reuse to Address Complex and Emerging Environmental Health Problems

Environmental health sciences (EHS) span many diverse disciplines. Within the EHS community, the National Institute of Environmental Health Sciences Superfund Research Program (SRP) funds multidisciplinary research aimed to address pressing and complex issues on how people are exposed to hazardous substances and their related health consequences with the goal of identifying strategies to reduce exposures and protect human health. While disentangling the interrelationships that contribute to environmental exposures and their effects on human health over the course of life remains difficult, advances in data science and data sharing offer a path forward to explore data across disciplines to reveal new insights. Multidisciplinary SRP-funded teams are well-positioned to examine how to best integrate EHS data across diverse research domains to address multifaceted environmental health problems. As such, SRP supported collaborative research projects designed to foster and enhance the interoperability and reuse of diverse and complex data streams. This perspective synthesizes those experiences as a landscape view of the challenges identified while working to increase the FAIR-ness (Findable, Accessible, Interoperable, and Reusable) of EHS data and opportunities to address them.

Perceptions and Opinions Towards Data-Sharing: A Survey of Addiction Journal Editorial Board Members

Background

We surveyed addiction journal editorial board members to better understand their opinions towards data-sharing.

Methods

Survey items consisted of Likert-type (e.g., one to five scale), multiple-choice, and free-response questions. Journal websites were searched for names and email addresses. Emails were distributed using SurveyMonkey. Descriptive statistics were used to characterize the responses.

Results

We received 178 responses (of 1039; 17.1%). Of these, 174 individuals agreed to participate in our study (97.8%). Most respondents did not know whether their journal had a data-sharing policy. Board members "somewhat agree" that addiction journals should recommend but not require data-sharing for submitted manuscripts [M=4.09 (SD=0.06); 95% CI: 3.97-4.22]. Items with the highest perceived benefit ratings were "secondary data use (e.g., meta-analysis)" [M=3.44 (SD=0.06); 95% CI: 3.31-3.56] and "increased transparency" [M=3.29 (SD=0.07); 95% CI: 3.14-3.43]. Items perceived to be the greatest barrier to data-sharing included "lack of metadata standards" [M=3.21 (SD=0.08); 95% CI: 3.06-3.36], "no incentive" [M=3.43 (SD=0.07); 95% CI: 3.30-3.57], "inadequate resources" [M=3.53 (SD=0.05); 95% CI: 3.42-3.63], and "protection of privacy"[M=3.22 (SD=0.07); 95% CI: 3.07-3.36].

Conclusion

Our results suggest addiction journal editorial board members believe data-sharing has a level of importance within the research community. However, most board members are unaware of their journals' data-sharing policies, and most data-sharing should be recommended but not required. Future efforts aimed at better understanding common reservations and benefits towards data-sharing, as well as avenues to optimize data-sharing while minimizing potential risks, are warranted.

Mediating factors of statistics anxiety in university students: a systematic review and meta-analysis

Statistics plays a key role in many areas of modern society, including technology, social and behavior studies, economics, and the sciences. Statistics anxiety (SA) has a detrimental impact on academic experiences in university populations, although the mediating factors remain underexplored. We conducted the first systematic review and meta-analysis focused on SA in university students in the context of statistical performance, individual differences in statistical learning, self-perceptions regarding the statistics course and instructor, and sociodemographic factors. Searches were carried out in the PsycINFO, PubMed, Scielo, and Web of Science databases according to our preregistration. Forty studies were selected for systematic review. Seventeen were included in a series of six meta-analyses concerning academic achievement, attitudes, self-perception, procrastination, and gender. The findings reveal learning strategies, procrastination, self-efficacy, and self-awareness as predictors of SA. However, the impact of sociodemographic data in these moderators is still uncharted. We conclude with a critical appraisal of the selected studies and present future directions for research in SA.

Open science practices for eating disorders research

This editorial seeks to encourage the increased application of three open science practices in eating disorders research: Preregistration, Registered Reports, and the sharing of materials, data, and code. For each of these practices, we introduce updated International Journal of Eating Disorders author and reviewer guidance. Updates include the introduction of open science badges; specific instructions about how to improve transparency; and the introduction of Registered Reports of systematic or meta-analytical reviews. The editorial also seeks to encourage the study of open science practices. Open science practices pose considerable time and other resource burdens. Therefore, research is needed to help determine the value of these added burdens and to identify efficient strategies for implementing open science practices.

Psychological Science in the Wake of COVID-19: Social, Methodological, and Metascientific Considerations

The COVID-19 pandemic has extensively changed the state of psychological science from what research questions psychologists can ask to which methodologies psychologists can use to investigate them. In this article, we offer a perspective on how to optimize new research in the pandemic's wake. Because this pandemic is inherently a social phenomenon-an event that hinges on human-to-human contact-we focus on socially relevant subfields of psychology. We highlight specific psychological phenomena that have likely shifted as a result of the pandemic and discuss theoretical, methodological, and practical considerations of conducting research on these phenomena. After this discussion, we evaluate metascientific issues that have been amplified by the pandemic. We aim to demonstrate how theoretically grounded views on the COVID-19 pandemic can help make psychological science stronger-not weaker-in its wake.

An empirical study of research ethics and their role in psychologists’ data sharing intentions using consequentialism theory of ethics

The purpose of this study is to examine how different ethical dimensions of egoism, utilitarianism, and deontology all help in the formation of psychologists’ research ethics for data sharing, and how the research ethics eventually affect psychologists making decisions regarding whether to engage in data sharing. This research utilized consequentialism theory of ethics as its theoretical framework to develop its research model of psychologists’ data sharing as mediated by research ethics. It conducted an online survey with psychologists in US academic institutions and collected a total of 362 valid responses. Then, it employed the structural equation modeling technique to evaluate the research model and related hypotheses of psychologists’ data sharing intentions as mediated by the profession’s research ethics. This research found that perceived career benefit, perceived community benefit, and norm of data sharing all significantly contribute to the formation of psychologists’ research ethics for data sharing, and then these research ethics, along with perceived community benefit and norm of data sharing, significantly influence psychologists’ data sharing intentions. This study suggests that the consequentialism theory of ethics nicely explains psychologists’ formation of their research ethics for data sharing and their decision to engage in data sharing. The study also suggests that research communities can better promote researchers’ data sharing behaviors by stimulating their research ethics through different ethical dimensions, including egoism (career benefit), utilitarianism (community benefit), and deontology (norm of data sharing).

Sorting the File Drawer: A Typology for Describing Unpublished Studies

A typology of unpublished studies is presented to describe various types of unpublished studies and the reasons for their nonpublication. Reasons for nonpublication are classified by whether they stem from an awareness of the study results (result-dependent reasons) or not (result-independent reasons) and whether the reasons affect the publication decisions of individual researchers or reviewers/editors. I argue that result-independent reasons for nonpublication are less likely to introduce motivated reasoning into the publication decision process than are result-dependent reasons. I also argue that some reasons for nonpublication would produce beneficial as opposed to problematic publication bias. The typology of unpublished studies provides a descriptive scheme that can facilitate understanding of the population of study results across the field of psychology, within subdisciplines of psychology, or within specific psychology research domains. The typology also offers insight into different publication biases and research-dissemination practices and can guide individual researchers in organizing their own file drawers of unpublished studies.

Online and Face-to-Face Performance on Two Cognitive Tasks in Children With Williams Syndrome

There has been an increase in cognitive assessment via the Internet, especially since the coronavirus disease 2019 surged the need for remote psychological assessment. This is the first study to investigate the appropriability of conducting cognitive assessments online with children with a neurodevelopmental condition and intellectual disability, namely, Williams syndrome. This study compared Raven’s Colored Progressive Matrices (RCPM) and British Picture Vocabulary Scale (BPVS) scores from two different groups of children with WS age 10–11 years who were assessed online (n = 14) or face-to-face (RCPM n = 12; BPVS n = 24). Bayesian t-tests showed that children’s RCPM scores were similar across testing conditions, but suggested BPVS scores were higher for participants assessed online. The differences between task protocols are discussed in line with these findings, as well as the implications for neurodevelopmental research.

PsyCuraDat: Designing a User-Oriented Curation Standard for Behavioral Psychological Research Data

Starting from the observation that data sharing in general and sharing of reusable behavioral data in particular is still scarce in psychology, we set out to develop a curation standard for behavioral psychological research data rendering data reuse more effective and efficient. Specifically, we propose a standard that is oriented toward the requirements of the psychological research process, thus considering the needs of researchers in their role as data providers and data users. To this end, we suggest that researchers should describe their data on three documentation levels reflecting researchers’ central decisions during the research process. In particular, these levels describe researchers’ decisions on the concrete research design that is most suitable to address the corresponding research question, its operationalization as well as a precise description of the subsequent data collection and analysis process. Accordingly, the first documentation level represents, for instance, researchers’ decision on the concrete hypotheses, inclusion/exclusion criteria and the number of measurement points as well as a conceptual presentation of all substantial variables included in the design. On the second level these substantial variables are presented within an extended codebook allowing for the linkage between the conceptual research design and the actually operationalized variables as presented within the data. Finally, the third level includes all materials, data preparation and analyses scripts as well as a detailed procedure graphic that allows the data user to link the information from all three documentation levels at a single glance. After a comprehensive presentation of the standard, we will offer some arguments for its integration into the psychological research process.

Does open data boost journal impact: evidence from Chinese economics

To encourage research transparency and replication, more and more journals have been requiring authors to share original datasets and analytic procedures supporting their publications. Does open data boost journal impact? In this article, we report one of the first empirical studies to assess the effects of open data on journal impact. China Industrial Economics (CIE) mandated authors to open their research data in the end of 2016, which is the first to embrace open data among Chinese journals and provides a natural experiment for policy evaluation. We use the data of 37 Chinese economics journals from 2001 to 2019 and apply synthetic control method to causally estimate the effects of open data, and our results show that open data has significantly increased the citations of journal articles. On average, the current- and second-year citations of articles published with CIE have increased by 1 ~ 4 times, and articles published before the open data policy also benefited from the spillover effect. Our findings suggest that journals can leverage compulsory open data to develop reputation and amplify academic impacts.

Optimizing the scientific study of suicide with open and transparent research practices

Suicide research is vitally important, yet-like psychology research more broadly-faces methodological challenges. In recent years, researchers have raised concerns about standard practices in psychological research, concerns that apply to suicide research and raise questions about its robustness and validity. In the present paper, we review these concerns and the corresponding solutions put forth by the "open science" community. These include using open science platforms, pre-registering studies, ensuring reproducible analyses, using high-powered studies, ensuring open access to research materials and products, and conducting replication studies. We build upon existing guides, address specific obstacles faced by suicide researchers, and offer a clear set of recommended practices for suicide researchers. In particular, we consider challenges that suicide researchers may face in seeking to adopt "open science" practices (e.g., prioritizing large samples) and suggest possible strategies that the field may use in order to ensure robust and transparent research, despite these challenges.

Publication rates from biomedical and behavioral and social science R01s funded by the National Institutes of Health

Prior research has shown a serious lack of research transparency resulting from the failure to publish study results in a timely manner. The National Institutes of Health (NIH) has increased its use of publication rate and time to publication as metrics for grant productivity. In this study, we analyze the publications associated with all R01 and U01 grants funded from 2008 through 2014, providing sufficient time for these grants to publish their findings, and identify predictors of time to publication based on a number of variables, including if a grant was coded as a behavioral and social sciences research (BSSR) grant or not. Overall, 2.4% of the 27,016 R01 and U01 grants did not have a publication associated with the grant within 60 months of the project start date, and this rate of zero publications was higher for BSSR grants (4.6%) than for non-BSSR grants (1.9%). Mean time in months to first publication was 15.2 months, longer for BSSR grants (22.4 months) than non-BSSR grants (13.6 months). Survival curves showed a more rapid reduction of risk to publish from non-BSSR vs BSSR grants. Cox regression models showed that human research (vs. animal, neither, or both) and clinical trials research (vs. not) are the strongest predictors of time to publication and failure to publish, but even after accounting for these and other predictors, BSSR grants continued to show longer times to first publication and greater risk of no publications than non-BSSR grants. These findings indicate that even with liberal criteria for publication (any publication associated with a grant), a small percentage of R01 and U01 grantees fail to publish in a timely manner, and that a number of factors, including human research, clinical trial research, child research, not being an early stage investigator, and conducting behavioral and social sciences research increase the risk of time to first publication.

Opening Pandora's Box: Peeking inside Psychology's data sharing practices, and seven recommendations for change

Open data-sharing is a valuable practice that ought to enhance the impact, reach, and transparency of a research project. While widely advocated by many researchers and mandated by some journals and funding agencies, little is known about detailed practices across psychological science. In a pre-registered study, we show that overall, few research papers directly link to available data in many, though not all, journals. Most importantly, even where open data can be identified, the majority of these lacked completeness and reusability—conclusions that closely mirror those reported outside of Psychology. Exploring the reasons behind these findings, we offer seven specific recommendations for engineering and incentivizing improved practices, so that the potential of open data can be better realized across psychology and social science more generally.

Dispersion matters: Diagnostics and control data computer simulation in Concealed Information Test studies

Binary classification has numerous applications. For one, lie detection methods typically aim to classify each tested person either as "liar" or as "truthteller" based on the given test results. To infer practical implications, as well as to compare different methods, it is essential to assess the diagnostic efficiency, such as demonstrating the number of correctly classified persons. However, this is not always straightforward. In Concealed Information Tests (CITs), the key predictor value (probe-irrelevant difference) for "truthtellers" is always similar (zero on average), and "liars" are always distinguished by a larger value (i.e., a larger number resulting from the CIT test, as compared to the zero baseline). Thereby, in general, the larger predictor values a given CIT method obtains for "liars" on average, the better this method is assumed to be. This has indeed been assumed in countless studies, and therefore, when comparing the classification efficiencies of two different designs, the mean difference of "liar" predictor values in the two designs were simply compared to each other (hence not collecting "truthteller" data to spare resources). We show, based on the meta-data of 12 different experimental designs collected in response time-based CIT studies, that differences in dispersion (i.e., variance in the data, e.g. the extent of random deviations from the zero average in case of "truthtellers") can substantially influence classification efficiency-to the point that, in extreme cases, one design may even be superior in classification despite having a larger mean "liar" predictor value. However, we also introduce a computer simulation procedure to estimate classification efficiency in the absence of "truthteller" data, and validate this procedure via a meta-analysis comparing outcomes based on empirical data versus simulated data.

Sharing and organizing research products as R packages

A consensus on the importance of open data and reproducible code is emerging. How should data and code be shared to maximize the key desiderata of reproducibility, permanence, and accessibility? Research assets should be stored persistently in formats that are not software restrictive, and documented so that others can reproduce and extend the required computations. The sharing method should be easy to adopt by already busy researchers. We suggest the R package standard as a solution for creating, curating, and communicating research assets. The R package standard, with extensions discussed herein, provides a format for assets and metadata that satisfies the above desiderata, facilitates reproducibility, open access, and sharing of materials through online platforms like GitHub and Open Science Framework. We discuss a stack of R resources that help users create reproducible collections of research assets, from experiments to manuscripts, in the RStudio interface. We created an R package, vertical, to help researchers incorporate these tools into their workflows, and discuss its functionality at length in an online supplement. Together, these tools may increase the reproducibility and openness of psychological science.

Quantity Over Quality? Reproducible Psychological Science from a Mixed Methods Perspective

A robust dialogue about the (un)reliability of psychological science findings has emerged in recent years. In response, metascience researchers have developed innovative tools to increase rigor, transparency, and reproducibility, stimulating rapid improvement and adoption of open science practices. However, existing reproducibility guidelines are geared toward purely quantitative study designs. This leaves some ambiguity as to how such guidelines should be implemented in mixed methods (MM) studies, which combine quantitative and qualitative research. Drawing on extant literature, our own experiences, and feedback from 79 self-identified MM researchers, the current paper addresses two main questions: (a) how and to what extent do existing reproducibility guidelines apply to MM study designs; and (b) can existing reproducibility guidelines be improved by incorporating best practices from qualitative research and epistemology? In answer, we offer 10 key recommendations for use within and outside of MM research. Finally, we argue that good science and good ethical practice are mutually reinforcing and lead to meaningful, credible science.

Embracing data preservation, sharing, and re-use in traumatic stress research

This editorial argues that it is time for the traumatic stress field to join the growing international movement towards Findable, Accessible, Interoperable, and Re-usable (FAIR) research data, and that we are well-positioned to do so. The field has a huge, largely untapped resource in the enormous number of rich potentially re-usable datasets that are not currently shared or preserved. We have several promising shared data resources created via international collaborative efforts by traumatic stress researchers, but we do not yet have common standards for data description, sharing, or preservation. And, despite the promise of novel findings from data sharing and re-use, there are a number of barriers to researchers' adoption of FAIR data practices. We present a vision for the future of FAIR traumatic stress data, and a call to action for the traumatic stress research community and individual researchers and research teams to help achieve this vision.

The Confidence Database

Understanding how people rate their confidence is critical for the characterization of a wide range of perceptual, memory, motor and cognitive processes. To enable the continued exploration of these processes, we created a large database of confidence studies spanning a broad set of paradigms, participant populations and fields of study. The data from each study are structured in a common, easy-to-use format that can be easily imported and analysed using multiple software packages. Each dataset is accompanied by an explanation regarding the nature of the collected data. At the time of publication, the Confidence Database (which is available at https://osf.io/s46pr/) contained 145 datasets with data from more than 8,700 participants and almost 4 million trials. The database will remain open for new submissions indefinitely and is expected to continue to grow. Here we show the usefulness of this large collection of datasets in four different analyses that provide precise estimations of several foundational confidence-related effects.

Did awarding badges increase data sharing in BMJ Open? A randomized controlled trial

Sharing data and code are important components of reproducible research. Data sharing in research is widely discussed in the literature; however, there are no well-established evidence-based incentives that reward data sharing, nor randomized studies that demonstrate the effectiveness of data sharing policies at increasing data sharing. A simple incentive, such as an Open Data Badge, might provide the change needed to increase data sharing in health and medical research. This study was a parallel group randomized controlled trial (protocol registration: doi:10.17605/OSF.IO/PXWZQ) with two groups, control and intervention, with 80 research articles published in BMJ Open per group, with a total of 160 research articles. The intervention group received an email offer for an Open Data Badge if they shared their data along with their final publication and the control group received an email with no offer of a badge if they shared their data with their final publication. The primary outcome was the data sharing rate. Badges did not noticeably motivate researchers who published in BMJ Open to share their data; the odds of awarding badges were nearly equal in the intervention and control groups (odds ratio = 0.9, 95% CI [0.1, 9.0]). Data sharing rates were low in both groups, with just two datasets shared in each of the intervention and control groups. The global movement towards open science has made significant gains with the development of numerous data sharing policies and tools. What remains to be established is an effective incentive that motivates researchers to take up such tools to share their data.

Do Readers Integrate Phonological Codes Across Saccades? A Bayesian Meta-Analysis and a Survey of the Unpublished Literature

It is commonly accepted that phonological codes can be activated parafoveally during reading and later used to aid foveal word recognition- a finding known as the phonological preview benefit. However, a closer look at the literature shows that this effect may be less consistent than what is sometimes believed. To determine the extent to which phonology is processed parafoveally, a Bayesian meta-analysis of 27 experiments and a survey of the unpublished literature were conducted. While the results were generally consistent with the phonological preview benefit (>90% probability of a true effect in gaze durations), the size of the effect was small. Readers of alphabetical orthographies obtained a modest benefit of only 4 ms in gaze durations. Interestingly, Chinese readers showed a larger effect (6-14 ms in size). There was no difference in the magnitude of the phonological preview benefit between homophone and pseudo-homophone previews, thus suggesting that the modest processing advantage is indeed related to the activation of phonological codes from the parafoveal word. Simulations revealed that the results are relatively robust to missing studies, although the effects may be 19-22% smaller if all missing studies found a null effect. The results suggest that while phonology can be processed parafoveally, this happens only to a limited extent. Because phonological priming effects in single-word recognition are small (10-13 ms; Rastle & Brysbaert, 2006) and there is a loss of visual acuity in the parafovea, it is argued that large phonological preview benefit effects may be unlikely.

Why We Need a Formal Systematic Approach to Validating Psychological Tests: The Case of the Rorschach Comprehensive System

This article documents and discusses the importance of using a formal systematic approach to validating psychological tests. To illustrate, results are presented from a systematic review of the validity findings cited in the Rorschach Comprehensive System (CS; Exner, 2003) test manual, originally conducted during the manuscript review process for Mihura, Meyer, Dumitrascu, and Bombel's (2013) CS meta-analyses. Our review documents (a) the degree to which the CS test manual reports validity findings for each test variable, (b) whether these findings are publicly accessible or unpublished studies coordinated by the test developer, and (c) the presence and nature of data discrepancies between the CS test manual and the cited source. Implications are discussed for the CS in particular, the Rorschach more generally, and psychological tests more broadly. Notably, a history of intensive scrutiny of the Rorschach has resulted in more stringent standards applied to it, even though its scales have more published and supportive construct validity meta-analyses than any other psychological test. Calls are made for (a) a mechanism to correct data errors in the scientific literature, (b) guidelines for test developers' key unpublished studies, and (c) systematic reviews and meta-analyses to become standard practice for all psychological tests.