Opening Pandora's Box: Peeking inside Psychology's data sharing practices, and seven recommendations for change

Promoting Data Sharing: The Moral Obligations of Public Funding Agencies

Sharing research data has great potential to benefit science and society. However, data sharing is still not common practice. Since public research funding agencies have a particular impact on research and researchers, the question arises: Are public funding agencies morally obligated to promote data sharing? We argue from a research ethics perspective that public funding agencies have several pro tanto obligations requiring them to promote data sharing. However, there are also pro tanto obligations that speak against promoting data sharing in general as well as with regard to particular instruments of such promotion. We examine and weigh these obligations and conclude that all things considered funders ought to promote the sharing of data. Even the instrument of mandatory data sharing policies can be justified under certain conditions.

Expanding the data Ark: an attempt to make the data from highly cited social science papers publicly available

Access to scientific data can enable independent reuse and verification; however, most data are not available and become increasingly irrecoverable over time. This study aimed to retrieve and preserve important datasets from 160 of the most highly-cited social science articles published between 2008-2013 and 2015-2018. We asked authors if they would share data in a public repository-the Data Ark-or provide reasons if data could not be shared. Of the 160 articles, data for 117 (73%, 95% CI [67%-80%]) were not available and data for 7 (4%, 95% CI [0%-12%]) were available with restrictions. Data for 36 (22%, 95% CI [16%-30%]) articles were available in unrestricted form: 29 of these datasets were already available and 7 datasets were made available in the Data Ark. Most authors did not respond to our data requests and a minority shared reasons for not sharing, such as legal or ethical constraints. These findings highlight an unresolved need to preserve important scientific datasets and increase their accessibility to the scientific community.

FAB: A "Dummy's" program for self-paced forward and backward reading

The self-paced reading paradigm has been popular and widely used in psycholinguistic research for several decades. The tool described in this paper, FAB (Forward and Backward reading), is a tool created to hopefully and maximally reduce the coding demands and simplify the operation costs for experimental researchers and clinical researchers who are doing experimental work, in their designing, coding, implementing, and analyzing self-paced reading tasks. Its basis in web languages (HTML, JavaScript) also promotes experimental implementation and material sharing in our era of open science. In addition, FAB has a unique forward-and-backward mode that can track regressive-like behaviors that are usually only recordable using eye-tracking or mouse-tracking equipment. In this paper, the specific application and usage of FAB is demonstrated in one laboratory and two online validation experiments. We hope this free and open-sourced tool can benefit research in a diverse range of contexts where self-paced reading is desirable.

Prevalence and predictors of data and code sharing in the medical and health sciences: systematic review with meta-analysis of individual participant data

OBJECTIVES

To synthesise research investigating data and code sharing in medicine and health to establish an accurate representation of the prevalence of sharing, how this frequency has changed over time, and what factors influence availability.

DESIGN

Systematic review with meta-analysis of individual participant data.

DATA SOURCES

Ovid Medline, Ovid Embase, and the preprint servers medRxiv, bioRxiv, and MetaArXiv were searched from inception to 1 July 2021. Forward citation searches were also performed on 30 August 2022.

REVIEW METHODS

Meta-research studies that investigated data or code sharing across a sample of scientific articles presenting original medical and health research were identified. Two authors screened records, assessed the risk of bias, and extracted summary data from study reports when individual participant data could not be retrieved. Key outcomes of interest were the prevalence of statements that declared that data or code were publicly or privately available (declared availability) and the success rates of retrieving these products (actual availability). The associations between data and code availability and several factors (eg, journal policy, type of data, trial design, and human participants) were also examined. A two stage approach to meta-analysis of individual participant data was performed, with proportions and risk ratios pooled with the Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis.

RESULTS

The review included 105 meta-research studies examining 2 121 580 articles across 31 specialties. Eligible studies examined a median of 195 primary articles (interquartile range 113-475), with a median publication year of 2015 (interquartile range 2012-2018). Only eight studies (8%) were classified as having a low risk of bias. Meta-analyses showed a prevalence of declared and actual public data availability of 8% (95% confidence interval 5% to 11%) and 2% (1% to 3%), respectively, between 2016 and 2021. For public code sharing, both the prevalence of declared and actual availability were estimated to be <0.5% since 2016. Meta-regressions indicated that only declared public data sharing prevalence estimates have increased over time. Compliance with mandatory data sharing policies ranged from 0% to 100% across journals and varied by type of data. In contrast, success in privately obtaining data and code from authors historically ranged between 0% and 37% and 0% and 23%, respectively.

CONCLUSIONS

The review found that public code sharing was persistently low across medical research. Declarations of data sharing were also low, increasing over time, but did not always correspond to actual sharing of data. The effectiveness of mandatory data sharing policies varied substantially by journal and type of data, a finding that might be informative for policy makers when designing policies and allocating resources to audit compliance.

SYSTEMATIC REVIEW REGISTRATION

Open Science Framework doi:10.17605/OSF.IO/7SX8U.

Harmonizing PTSD severity scales across instruments and sites

OBJECTIVE

The variety of instruments used to assess posttraumatic stress disorder (PTSD) allows for flexibility, but also creates challenges for data synthesis. The objective of this work was to use a multisite mega analysis to derive quantitative recommendations for equating scores across measures of PTSD severity.

METHOD

Empirical Bayes harmonization and linear models were used to describe and mitigate site and covariate effects. Quadratic models for converting scores across PTSD assessments were constructed using bootstrapping and tested on hold out data.

RESULTS

We aggregated 17 data sources and compiled an n = 5,634 sample of individuals who were assessed for PTSD symptoms. We confirmed our hypothesis that harmonization and covariate adjustments would significantly improve inference of scores across instruments. Harmonization significantly reduced cross-dataset variance (28%, p < .001), and models for converting scores across instruments were well fit (median R² = 0.985) with an average root mean squared error of 1.46 on sum scores.

CONCLUSIONS

These methods allow PTSD symptom severity to be placed on multiple scales and offers interesting empirical perspectives on the role of harmonization in the behavioral sciences. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Opportunities for improving data sharing and FAIR data practices to advance global mental health

It is crucial to optimize global mental health research to address the high burden of mental health challenges and mental illness for individuals and societies. Data sharing and reuse have demonstrated value for advancing science and accelerating knowledge development. The FAIR (Findable, Accessible, Interoperable, and Reusable) Guiding Principles for scientific data provide a framework to improve the transparency, efficiency, and impact of research. In this review, we describe ethical and equity considerations in data sharing and reuse, delineate the FAIR principles as they apply to mental health research, and consider the current state of FAIR data practices in global mental health research, identifying challenges and opportunities. We describe noteworthy examples of collaborative efforts, often across disciplinary and national boundaries, to improve Findability and Accessibility of global mental health data, as well as efforts to create integrated data resources and tools that improve Interoperability and Reusability. Based on this review, we suggest a vision for the future of FAIR global mental health research and suggest practical steps for researchers with regard to study planning, data preservation and indexing, machine-actionable metadata, data reuse to advance science and improve equity, metrics and recognition.

How to trust a scientist

Epistemic trust among scientists is inevitable. There are two questions about this: (1) What is the content of this trust, what do scientists trust each other for? (2) Is such trust epistemically justified? I argue that if we assume a traditional answer to (1), namely that scientists trust each other to be reliable informants, then the answer to question (2) is negative, certainly for the biomedical and social sciences. This motivates a different construal of trust among scientists and therefore a different answer to (1): scientists trust each other to only testify to claims that are backed by evidence gathered in accordance with prevailing methodological standards. On this answer, trust among scientists is epistemically justified.

Open data and data sharing in articles about COVID-19 published in preprint servers medRxiv and bioRxiv

This study aimed to analyze the content of data availability statements (DAS) and the actual sharing of raw data in preprint articles about COVID-19. The study combined a bibliometric analysis and a cross-sectional survey. We analyzed preprint articles on COVID-19 published on medRxiv and bioRxiv from January 1, 2020 to March 30, 2020. We extracted data sharing statements, tried to locate raw data when authors indicated they were available, and surveyed authors. The authors were surveyed in 2020–2021. We surveyed authors whose articles did not include DAS, who indicated that data are available on request, or their manuscript reported that raw data are available in the manuscript, but raw data were not found. Raw data collected in this study are published on Open Science Framework (https://osf.io/6ztec/). We analyzed 897 preprint articles. There were 699 (78%) articles with Data/Code field present on the website of a preprint server. In 234 (26%) preprints, data/code sharing statement was reported within the manuscript. For 283 preprints that reported that data were accessible, we found raw data/code for 133 (47%) of those 283 preprints (15% of all analyzed preprint articles). Most commonly, authors indicated that data were available on GitHub or another clearly specified web location, on (reasonable) request, in the manuscript or its supplementary files. In conclusion, preprint servers should require authors to provide data sharing statements that will be included both on the website and in the manuscript. Education of researchers about the meaning of data sharing is needed.

Measurement practices exacerbate the generalizability crisis: Novel digital measures can help

Psychology's tendency to focus on confirmatory analyses before ensuring constructs are clearly defined and accurately measured is exacerbating the generalizability crisis. Our growing use of digital behaviors as predictors has revealed the fragility of subjective measures and the latent constructs they scaffold. However, new technologies can provide opportunities to improve conceptualizations, theories, and measurement practices.