PsychoPy2: Experiments in behavior made easy

PsychoPy is an application for the creation of experiments in behavioral science (psychology, neuroscience, linguistics, etc.) with precise spatial control and timing of stimuli. It now provides a choice of interface; users can write scripts in Python if they choose, while those who prefer to construct experiments graphically can use the new Builder interface. Here we describe the features that have been added over the last 10 years of its development. The most notable addition has been that Builder interface, allowing users to create studies with minimal or no programming, while also allowing the insertion of Python code for maximal flexibility. We also present some of the other new features, including further stimulus options, asynchronous time-stamped hardware polling, and better support for open science and reproducibility. Tens of thousands of users now launch PsychoPy every month, and more than 90 people have contributed to the code. We discuss the current state of the project, as well as plans for the future.

Psychedelic microdosing benefits and challenges: an empirical codebook

BACKGROUND

Microdosing psychedelics is the practice of consuming very low, sub-hallucinogenic doses of a psychedelic substance, such as lysergic acid diethylamide (LSD) or psilocybin-containing mushrooms. According to media reports, microdosing has grown in popularity, yet the scientific literature contains minimal research on this practice. There has been limited reporting on adverse events associated with microdosing, and the experiences of microdosers in community samples have not been categorized.

METHODS

In the present study, we develop a codebook of microdosing benefits and challenges (MDBC) based on the qualitative reports of a real-world sample of 278 microdosers.

RESULTS

We describe novel findings, both in terms of beneficial outcomes, such as improved mood (26.6%) and focus (14.8%), and in terms of challenging outcomes, such as physiological discomfort (18.0%) and increased anxiety (6.7%). We also show parallels between benefits and drawbacks and discuss the implications of these results. We probe for substance-dependent differences, finding that psilocybin-only users report the benefits of microdosing were more important than other users report.

CONCLUSIONS

These mixed-methods results help summarize and frame the experiences reported by an active microdosing community as high-potential avenues for future scientific research. The MDBC taxonomy reported here informs future research, leveraging participant reports to distil the highest-potential intervention targets so research funding can be efficiently allocated. Microdosing research complements the full-dose literature as clinical treatments are developed and neuropharmacological mechanisms are sought. This framework aims to inform researchers and clinicians as experimental microdosing research begins in earnest in the years to come.

EECoG-Comp: An Open Source Platform for Concurrent EEG/ECoG Comparisons-Applications to Connectivity Studies

Electrophysiological Source Imaging (ESI) is hampered by lack of "gold standards" for model validation. Concurrent electroencephalography (EEG) and electrocorticography (ECoG) experiments (EECoG) are useful for this purpose, especially primate models due to their flexibility and translational value for human research. Unfortunately, there is only one EECoG experiments in the public domain that we know of: the Multidimensional Recording (MDR) is based on a single monkey ( www.neurotycho.org ). The mining of this type of data is hindered by lack of specialized procedures to deal with: (1) Severe EECoG artifacts due to the experimental produces; (2) Sophisticated forward models that account for surgery induced skull defects and implanted ECoG electrode strips; (3) Reliable statistical procedures to estimate and compare source connectivity (partial correlation). We provide solutions to the processing issues just mentioned with EECoG-Comp: an open source platform ( https://github.com/Vincent-wq/EECoG-Comp ). EECoG lead fields calculated with FEM (Simbio) for MDR data are also provided and were used in other papers of this special issue. As a use case with the MDR, we show: (1) For real MDR data, 4 popular ESI methods (MNE, LCMV, eLORETA and SSBL) showed significant but moderate concordance with a usual standard, the ECoG Laplacian (standard partial [Formula: see text]); (2) In both monkey and human simulations, all ESI methods as well as Laplacian had a significant but poor correspondence with the true source connectivity. These preliminary results may stimulate the development of improved ESI connectivity estimators but require the availability of more EECoG data sets to obtain neurobiologically valid inferences.

An open toolkit for tracking open science partnership implementation and impact

Serious concerns about the way research is organized collectively are increasingly being raised. They include the escalating costs of research and lower research productivity, low public trust in researchers to report the truth, lack of diversity, poor community engagement, ethical concerns over research practices, and irreproducibility. Open science (OS) collaborations comprise of a set of practices including open access publication, open data sharing and the absence of restrictive intellectual property rights with which institutions, firms, governments and communities are experimenting in order to overcome these concerns. We gathered two groups of international representatives from a large variety of stakeholders to construct a toolkit to guide and facilitate data collection about OS and non-OS collaborations. Ultimately, the toolkit will be used to assess and study the impact of OS collaborations on research and innovation. The toolkit contains the following four elements: 1) an annual report form of quantitative data to be completed by OS partnership administrators; 2) a series of semi-structured interview guides of stakeholders; 3) a survey form of participants in OS collaborations; and 4) a set of other quantitative measures best collected by other organizations, such as research foundations and governmental or intergovernmental agencies. We opened our toolkit to community comment and input. We present the resulting toolkit for use by government and philanthropic grantors, institutions, researchers and community organizations with the aim of measuring the implementation and impact of OS partnership across these organizations. We invite these and other stakeholders to not only measure, but to share the resulting data so that social scientists and policy makers can analyse the data across projects.

Scientific Integrity Principles and Best Practices: Recommendations from a Scientific Integrity Consortium

A Scientific Integrity Consortium developed a set of recommended principles and best practices that can be used broadly across scientific disciplines as a mechanism for consensus on scientific integrity standards and to better equip scientists to operate in a rapidly changing research environment. The two principles that represent the umbrella under which scientific processes should operate are as follows: (1) Foster a culture of integrity in the scientific process. (2) Evidence-based policy interests may have legitimate roles to play in influencing aspects of the research process, but those roles should not interfere with scientific integrity. The nine best practices for instilling scientific integrity in the implementation of these two overarching principles are (1) Require universal training in robust scientific methods, in the use of appropriate experimental design and statistics, and in responsible research practices for scientists at all levels, with the training content regularly updated and presented by qualified scientists. (2) Strengthen scientific integrity oversight and processes throughout the research continuum with a focus on training in ethics and conduct. (3) Encourage reproducibility of research through transparency. (4) Strive to establish open science as the standard operating procedure throughout the scientific enterprise. (5) Develop and implement educational tools to teach communication skills that uphold scientific integrity. (6) Strive to identify ways to further strengthen the peer review process. (7) Encourage scientific journals to publish unanticipated findings that meet standards of quality and scientific integrity. (8) Seek harmonization and implementation among journals of rapid, consistent, and transparent processes for correction and/or retraction of published papers. (9) Design rigorous and comprehensive evaluation criteria that recognize and reward the highest standards of integrity in scientific research.

Changing institutional incentives to foster sound scientific practices: One department

Replicable research and open science are of value to our field and to society at large, but most universities provide no incentives to adopt these practices. Instead, current incentive structures favor novel research, which has led to a situation in which few researchers take the time to do replications, share protocols, or share data. Obviously, several approaches to remedy this situation are possible. However, little progress can be made if becoming involved in such activities reduces a researcher's chances of rank and status advancement and other rewards. I describe in this article the way my department has modified our incentive structure to tackle this problem, including how the changes influence my research as a developmental psychologist. Finally, I offer suggestions for faculty who wish to initiate similar changes in their institutions.

Ecohydrological consequences of tree removal in an urban park evaluated using open data, free software and a minimalist measuring campaign

With ongoing global climate change and an increasingly urbanized population, the importance of city parks and other forms of urban vegetation increases. Trees in urban parks can play an important role in mitigating runoff and delivering other ecosystem services. Park managers, E-NGOs, citizen scientists and others are increasingly called upon to evaluate the possible consequences of changes in park management such as, e.g., tree removal. Here, we present an unorthodox approach to hydrological modelling and its potential use in local policy making regarding urban greenery. The approach consists of a minimalist field campaign to characterize vegetation and soil moisture status combined with a novel model calibration using freely available data and software. During modelling, we were able to obtain coefficients of determination (R²) of 0.66 and 0.73 for probe-measured and simulated soil moisture under tree stand and park lawn land covers respectively. The results demonstrated that tree cover had a significant positive effect on the hydrological regime of the locality through interception, transpiration and effects on soil moisture. Simulations suggested that tree cover was twice as effective at mitigating runoff than park lawn and almost seven times better than impervious surfaces. In the case of a potential replacement of tree vegetation in favour of park lawn or impervious surfaces an increase in runoff of 14% and 81% respectively could be expected. The main conclusion drawn from our study was that such an approach can be a very useful tool for supporting local decision-making processes as it offers a freely available, cheap and relatively easy-to-use way to describe the hydrological consequences of landcover change (e.g., tree removal) with sufficient accuracy.

Guidelines for open peer review implementation

Open peer review (OPR) is moving into the mainstream, but it is often poorly understood and surveys of researcher attitudes show important barriers to implementation. As more journals move to implement and experiment with the myriad of innovations covered by this term, there is a clear need for best practice guidelines to guide implementation. This brief article aims to address this knowledge gap, reporting work based on an interactive stakeholder workshop to create best-practice guidelines for editors and journals who wish to transition to OPR. Although the advice is aimed mainly at editors and publishers of scientific journals, since this is the area in which OPR is at its most mature, many of the principles may also be applicable for the implementation of OPR in other areas (e.g., books, conference submissions).

Is policy 'liberalization' associated with higher odds of adolescent cannabis use? A re-analysis of data from 38 countries

BACKGROUND

Policy makers worldwide face the choice of whether to reform cannabis policy from the 'full prohibition' model. A paper by Shi et al. (2015) suggested that such 'liberalization' is significantly associated with higher odds of adolescent cannabis use.

AIM

To test the validity and reliability of Shi et al's conclusion that the HBSC data show an association between policy 'liberalization' and increased likelihood of adolescent cannabis use.

METHODS

Replication and re-analysis of the same pooled data from three waves of the Health Behaviour in School-aged Children (HBSC) survey (2001/2, 2005/6 and 2009/10). This replicates - as far as possible - the coding and analytical strategy of the Shi et al article. The re-analysis makes some improvements by: excluding a variable ('number of siblings') for which many cases have missing data; including available data from the theoretically relevant case of Sweden for the latter two waves of the HBSC survey, which Shi et al omit; and including random slopes for gender between countries as well as random intercepts for countries in the mixed effects model, as the predictive effect of gender on cannabis use varies across countries.

RESULTS

Shi et al's verbal summary of their findings is not supported by detailed interpretation of their own numerical results. Without making the suggested amendments, it is possible to find a statistically significant association between policy 'liberalization' and higher odds of some measures of adolescent cannabis use. But when these improvements are made, this association becomes statistically non-significant.

CONCLUSION

Using a larger and more theoretically relevant sample of the HBSC respondents and an improved statistical model shows that the HBSC data do not reveal a statistically significant association between policy 'liberalization' and higher odds of adolescent cannabis use.

Textbooks authors, publishers, formats and costs in higher education

Objectives

There is little empirical data reported on retail prices of college textbooks beyond self-reported surveys and no published datasets. Textbooks, as an ancillary cost, can contribute to the overall rising cost of education which can impact upon students’ ability to succeed in Higher Education. This study sought to understand more about costs of college textbooks by conducting a systematic collection of several thousand textbooks from faculty readings lists in one Higher Education Institution in Ireland and a retrieval and analysis of the retail prices of a selection of those books.

Data description

Queries were made of the course catalogue database of a Higher Education Institution in Ireland resulting in generation of records for required and recommended textbooks for 15,414 books from 3030 unique courses for the academic year 2017–2018. This data was cleaned and processed before being used to query Google Books API. The dataset presented here represents the combination of data from the course catalogue and the Google Books API queries and comprises 2940 records of textbooks. Details for each book including title, authors, publisher, ISBN, retail price, ebook format, pdf availability, and public domain availability.

Publishers' Responsibilities in Promoting Data Quality and Reproducibility

Scholarly publishers can help to increase data quality and reproducible research by promoting transparency and openness. Increasing transparency can be achieved by publishers in six key areas: (1) understanding researchers' problems and motivations, by conducting and responding to the findings of surveys; (2) raising awareness of issues and encouraging behavioural and cultural change, by introducing consistent journal policies on sharing research data, code and materials; (3) improving the quality and objectivity of the peer-review process by implementing reporting guidelines and checklists and using technology to identify misconduct; (4) improving scholarly communication infrastructure with journals that publish all scientifically sound research, promoting study registration, partnering with data repositories and providing services that improve data sharing and data curation; (5) increasing incentives for practising open research with data journals and software journals and implementing data citation and badges for transparency; and (6) making research communication more open and accessible, with open-access publishing options, permitting text and data mining and sharing publisher data and metadata and through industry and community collaboration. This chapter describes practical approaches being taken by publishers, in these six areas, their progress and effectiveness and the implications for researchers publishing their work.

Ideation and implementation of an open science drug discovery business model - M4K Pharma

M4K Pharma was incorporated to launch an open science drug discovery program that relies on regulatory exclusivity as its primary intellectual property and commercial asset, in lieu of patents.In many cases and in key markets, using regulatory exclusivity can provide equivalent commercial protection to patents, while also being compatible with open science. The model is proving attractive to government, foundation and individual funders, who collectively have different expectations for returns on investment compared with biotech, pharmaceutical companies, or venture capital investors.In the absence of these investor-driven requirements for returns, it should be possible to commercialize therapeutics at affordable prices.M4K is piloting this open science business model in a rare paediatric brain tumour, but there is no reason it should not be more widely applicable.

The Bermuda Triangle: The Pragmatics, Policies, and Principles for Data Sharing in the History of the Human Genome Project

The Bermuda Principles for DNA sequence data sharing are an enduring legacy of the Human Genome Project (HGP). They were adopted by the HGP at a strategy meeting in Bermuda in February of 1996 and implemented in formal policies by early 1998, mandating daily release of HGP-funded DNA sequences into the public domain. The idea of daily sharing, we argue, emanated directly from strategies for large, goal-directed molecular biology projects first tested within the "community" of C. elegans researchers, and were introduced and defended for the HGP by the nematode biologists John Sulston and Robert Waterston. In the C. elegans community, and subsequently in the HGP, daily sharing served the pragmatic goals of quality control and project coordination. Yet in the HGP human genome, we also argue, the Bermuda Principles addressed concerns about gene patents impeding scientific advancement, and were aspirational and flexible in implementation and justification. They endured as an archetype for how rapid data sharing could be realized and rationalized, and permitted adaptation to the needs of various scientific communities. Yet in addition to the support of Sulston and Waterston, their adoption also depended on the clout of administrators at the US National Institutes of Health (NIH) and the UK nonprofit charity the Wellcome Trust, which together funded 90% of the HGP human sequencing effort. The other nations wishing to remain in the HGP consortium had to accommodate to the Bermuda Principles, requiring exceptions from incompatible existing or pending data access policies for publicly funded research in Germany, Japan, and France. We begin this story in 1963, with the biologist Sydney Brenner's proposal for a nematode research program at the Laboratory of Molecular Biology (LMB) at the University of Cambridge. We continue through 2003, with the completion of the HGP human reference genome, and conclude with observations about policy and the historiography of molecular biology.

Developing computational biology at meridian 23° E, and a little eastwards

Modern biology is experiencing a deep transformation by the expansion of molecular-level measurements at all scales, using omics technologies. A key element in this transformation is the field of bioinformatics, that has—in the meanwhile—permeated pretty much all of biological and biomedical research and is now emerging as a key inter-disciplinary area that connects the natural sciences, chemical and electrical engineering, science education and science policy, on a number of science and technology fronts. The strong tradition of open access for large volumes of raw data, collections of complex results and high-quality algorithm implementations in bioinformatics makes the field a unique, special case of open science. We report on our recent research activities, the development of training initiatives in the wider region during the past years, and the lessons learned regarding our efforts away from major epicenters, within the general context of open science.

Identifying the challenges in implementing open science [version 1; peer review: 2 approved]

Areas of open science (OS) policy and practice are already relatively well-advanced in several countries and sectors through the initiatives of some governments, funders, philanthropy, researchers and the community. Nevertheless, the current research and innovation system, including in the focus of this report, the life sciences, remains weighted against OS. In October 2017, thought-leaders from across the world gathered at an Open Science Leadership Forum in the Washington DC office of the Bill and Melinda Gates Foundation to share their views on what successful OS looks like. We focused on OS partnerships as this is an emerging model that aims to accelerate science and innovation. These outcomes are captured in a first meeting report: Defining Success in Open Science.

On several occasions, these conversations turned to the challenges that must be addressed and new policies required to effectively and sustainably advance OS practice. Thereupon, in this report, we describe the concerns raised and what is needed to address them supplemented by our review of the literature, and suggest the stakeholder groups that may be best placed to begin to take action. It emerges that to be successful, OS will require the active engagement of all stakeholders: while the research community must develop research questions, identify partners and networks, policy communities need to create an environment that is supportive of experimentation by removing barriers. This report aims to contribute to ongoing discussions about OS and its implementation. It is also part of a step-wise process to develop and mobilize a toolkit of quantitative and qualitative indicators to assist global stakeholders in implementing high value OS collaborations. Currently in co-development through an open and international process, this set of measures will allow the generation of needed evidence on the influence of OS partnerships on research, innovation, and critical social and economic goals.

Photo images, 3D models and CT scanned data of loaches (Botiidae, Cobitidae and Nemacheilidae) of Japan

Background

Loach is one of the major cypriniform fishes in freshwater habitats of Japan; 35 taxa/clades have, until now, been recognised. Parallel to genetic studies, morphological examinations are needed for further development of loach study, eventually ichthyology and fish biology. Digital archiving, concerning taxonomy, ecology, ethology etc., is one of the progressive challenges for the open science of biology. This paper aimed to online publish photo images, 3D models and CT scanned data of all the known clades of loaches inhabiting Japan (103 individuals in total with several type specimens), contributing to ichthyology and public interest of biodiversity/biology.

New information

Photo images, 3D models and CT scanned data of all the known 35 taxa/clades of loaches inhabiting in Japan were online published at http://ffish.asia/loachesOfJapan and http://ffish.asia/loachesOfJapan3D.

Reproducible research and GIScience: an evaluation using AGILE conference papers

The demand for reproducible research is on the rise in disciplines concerned with data analysis and computational methods. Therefore, we reviewed current recommendations for reproducible research and translated them into criteria for assessing the reproducibility of articles in the field of geographic information science (GIScience). Using this criteria, we assessed a sample of GIScience studies from the Association of Geographic Information Laboratories in Europe (AGILE) conference series, and we collected feedback about the assessment from the study authors. Results from the author feedback indicate that although authors support the concept of performing reproducible research, the incentives for doing this in practice are too small. Therefore, we propose concrete actions for individual researchers and the GIScience conference series to improve transparency and reproducibility. For example, to support researchers in producing reproducible work, the GIScience conference series could offer awards and paper badges, provide author guidelines for computational research, and publish articles in Open Access formats.

A guideline for reporting experimental protocols in life sciences

Experimental protocols are key when planning, performing and publishing research in many disciplines, especially in relation to the reporting of materials and methods. However, they vary in their content, structure and associated data elements. This article presents a guideline for describing key content for reporting experimental protocols in the domain of life sciences, together with the methodology followed in order to develop such guideline. As part of our work, we propose a checklist that contains 17 data elements that we consider fundamental to facilitate the execution of the protocol. These data elements are formally described in the SMART Protocols ontology. By providing guidance for the key content to be reported, we aim (1) to make it easier for authors to report experimental protocols with necessary and sufficient information that allow others to reproduce an experiment, (2) to promote consistency across laboratories by delivering an adaptable set of data elements, and (3) to make it easier for reviewers and editors to measure the quality of submitted manuscripts against an established criteria. Our checklist focuses on the content, what should be included. Rather than advocating a specific format for protocols in life sciences, the checklist includes a full description of the key data elements that facilitate the execution of the protocol.

Biochem-Env: a platform of biochemistry for research in environmental and agricultural sciences

Biochemical indicators are potent tools to assess ecosystem functioning under anthropic and global pressures. Nevertheless, additional work is needed to improve the methods used for the measurement of these indicators, and for a more relevant interpretation of the obtained results. To face these challenges, the platform Biochem-Env aims at providing innovative and standardized measurement protocols, as well as database and information system favoring result interpretation and opening. Its skills and tools are also offered for expertise, consulting, training, and standardization. In addition, the platform is a service of a French Research Infrastructure for Analysis and Experimentation on Ecosystems, for research in environmental and agricultural sciences.

The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles

Despite growing interest in Open Access (OA) to scholarly literature, there is an unmet need for large-scale, up-to-date, and reproducible studies assessing the prevalence and characteristics of OA. We address this need using oaDOI, an open online service that determines OA status for 67 million articles. We use three samples, each of 100,000 articles, to investigate OA in three populations: (1) all journal articles assigned a Crossref DOI, (2) recent journal articles indexed in Web of Science, and (3) articles viewed by users of Unpaywall, an open-source browser extension that lets users find OA articles using oaDOI. We estimate that at least 28% of the scholarly literature is OA (19M in total) and that this proportion is growing, driven particularly by growth in Gold and Hybrid. The most recent year analyzed (2015) also has the highest percentage of OA (45%). Because of this growth, and the fact that readers disproportionately access newer articles, we find that Unpaywall users encounter OA quite frequently: 47% of articles they view are OA. Notably, the most common mechanism for OA is not Gold, Green, or Hybrid OA, but rather an under-discussed category we dub Bronze: articles made free-to-read on the publisher website, without an explicit Open license. We also examine the citation impact of OA articles, corroborating the so-called open-access citation advantage: accounting for age and discipline, OA articles receive 18% more citations than average, an effect driven primarily by Green and Hybrid OA. We encourage further research using the free oaDOI service, as a way to inform OA policy and practice.

Sustainable computational science: the ReScience initiative

Computer science offers a large set of tools for prototyping, writing, running, testing, validating, sharing and reproducing results; however, computational science lags behind. In the best case, authors may provide their source code as a compressed archive and they may feel confident their research is reproducible. But this is not exactly true. James Buckheit and David Donoho proposed more than two decades ago that an article about computational results is advertising, not scholarship. The actual scholarship is the full software environment, code, and data that produced the result. This implies new workflows, in particular in peer-reviews. Existing journals have been slow to adapt: source codes are rarely requested and are hardly ever actually executed to check that they produce the results advertised in the article. ReScience is a peer-reviewed journal that targets computational research and encourages the explicit replication of already published research, promoting new and open-source implementations in order to ensure that the original research can be replicated from its description. To achieve this goal, the whole publishing chain is radically different from other traditional scientific journals. ReScience resides on GitHub where each new implementation of a computational study is made available together with comments, explanations, and software tests.

Finding the traces of behavioral and cognitive processes in big data and naturally occurring datasets

Today, people generate and store more data than ever before as they interact with both real and virtual environments. These digital traces of behavior and cognition offer cognitive scientists and psychologists an unprecedented opportunity to test theories outside the laboratory. Despite general excitement about big data and naturally occurring datasets among researchers, three "gaps" stand in the way of their wider adoption in theory-driven research: the imagination gap, the skills gap, and the culture gap. We outline an approach to bridging these three gaps while respecting our responsibilities to the public as participants in and consumers of the resulting research. To that end, we introduce Data on the Mind ( http://www.dataonthemind.org ), a community-focused initiative aimed at meeting the unprecedented challenges and opportunities of theory-driven research with big data and naturally occurring datasets. We argue that big data and naturally occurring datasets are most powerfully used to supplement-not supplant-traditional experimental paradigms in order to understand human behavior and cognition, and we highlight emerging ethical issues related to the collection, sharing, and use of these powerful datasets.

The what, why, and how of born-open data

Although many researchers agree that scientific data should be open to scrutiny to ferret out poor analyses and outright fraud, most raw data sets are not available on demand. There are many reasons researchers do not open their data, and one is technical. It is often time consuming to prepare and archive data. In response, my laboratory has automated the process such that our data are archived the night they are created without any human approval or action. All data are versioned, logged, time stamped, and uploaded including aborted runs and data from pilot subjects. The archive is GitHub, github.com, the world's largest collection of open-source materials. Data archived in this manner are called born open. In this paper, I discuss the benefits of born-open data and provide a brief technical overview of the process. I also address some of the common concerns about opening data before publication.

Towards a new generation of agricultural system data, models and knowledge products: Information and communication technology

Agricultural modeling has long suffered from fragmentation in model implementation. Many models are developed, there is much redundancy, models are often poorly coupled, model component re-use is rare, and it is frequently difficult to apply models to generate real solutions for the agricultural sector. To improve this situation, we argue that an open, self-sustained, and committed community is required to co-develop agricultural models and associated data and tools as a common resource. Such a community can benefit from recent developments in information and communications technology (ICT). We examine how such developments can be leveraged to design and implement the next generation of data, models, and decision support tools for agricultural production systems. Our objective is to assess relevant technologies for their maturity, expected development, and potential to benefit the agricultural modeling community. The technologies considered encompass methods for collaborative development and for involving stakeholders and users in development in a transdisciplinary manner. Our qualitative evaluation suggests that as an overall research challenge, the interoperability of data sources, modular granular open models, reference data sets for applications and specific user requirements analysis methodologies need to be addressed to allow agricultural modeling to enter in the big data era. This will enable much higher analytical capacities and the integrated use of new data sources. Overall agricultural systems modeling needs to rapidly adopt and absorb state-of-the-art data and ICT technologies with a focus on the needs of beneficiaries and on facilitating those who develop applications of their models. This adoption requires the widespread uptake of a set of best practices as standard operating procedures.

psiTurk: An open-source framework for conducting replicable behavioral experiments online

Online data collection has begun to revolutionize the behavioral sciences. However, conducting carefully controlled behavioral experiments online introduces a number of new of technical and scientific challenges. The project described in this paper, psiTurk, is an open-source platform which helps researchers develop experiment designs which can be conducted over the Internet. The tool primarily interfaces with Amazon's Mechanical Turk, a popular crowd-sourcing labor market. This paper describes the basic architecture of the system and introduces new users to the overall goals. psiTurk aims to reduce the technical hurdles for researchers developing online experiments while improving the transparency and collaborative nature of the behavioral sciences.