Working through some issues

Incentives for Research Effort: An Evolutionary Model of Publication Markets with Double-Blind and Open Review

Contemporary debates about scientific institutions and practice feature many proposed reforms. Most of these require increased efforts from scientists. But how do scientists' incentives for effort interact? How can scientific institutions encourage scientists to invest effort in research? We explore these questions using a game-theoretic model of publication markets. We employ a base game between authors and reviewers, before assessing some of its tendencies by means of analysis and simulations. We compare how the effort expenditures of these groups interact in our model under a variety of settings, such as double-blind and open review systems. We make a number of findings, including that open review can increase the effort of authors in a range of circumstances and that these effects can manifest in a policy-relevant period of time. However, we find that open review's impact on authors' efforts is sensitive to the strength of several other influences.

Can’t Make it Better nor Worse

Abstract. Some of the most popular psychological questionnaires violate general rules of item construction: precise, positively keyed items without negations, multiple aspects of content, absolute statements, or vague quantifiers. To investigate if following these rules results in more desirable psychometric properties, 1,733 participants completed online either the original NEO Five-Factor Inventory, an “improved” version whose items follow the rules of item construction, or a “deteriorated” version whose items strongly violate these rules. We compared reliability estimates, item-total correlations, Confirmatory Factor Analysis (CFA) model fit, and fit to the partial credit model between the three versions. Neither of the manipulations resulted in considerable or consistent effects on any of the psychometric indices. Our results question the ability of standard analyses in test construction to distinguish good items from bad ones, as well as the effectiveness of general rules of item construction. To increase the reproducibility of psychological science, more focus should be laid on improving psychological measures.

Handling Multiplicity in Neuroimaging Through Bayesian Lenses with Multilevel Modeling

Here we address the current issues of inefficiency and over-penalization in the massively univariate approach followed by the correction for multiple testing, and propose a more efficient model that pools and shares information among brain regions. Using Bayesian multilevel (BML) modeling, we control two types of error that are more relevant than the conventional false positive rate (FPR): incorrect sign (type S) and incorrect magnitude (type M). BML also aims to achieve two goals: 1) improving modeling efficiency by having one integrative model and thereby dissolving the multiple testing issue, and 2) turning the focus of conventional null hypothesis significant testing (NHST) on FPR into quality control by calibrating type S errors while maintaining a reasonable level of inference efficiency. The performance and validity of this approach are demonstrated through an application at the region of interest (ROI) level, with all the regions on an equal footing: unlike the current approaches under NHST, small regions are not disadvantaged simply because of their physical size. In addition, compared to the massively univariate approach, BML may simultaneously achieve increased spatial specificity and inference efficiency, and promote results reporting in totality and transparency. The benefits of BML are illustrated in performance and quality checking using an experimental dataset. The methodology also avoids the current practice of sharp and arbitrary thresholding in the p-value funnel to which the multidimensional data are reduced. The BML approach with its auxiliary tools is available as part of the AFNI suite for general use.

The Problem of False Discovery: Many Scientific Results Can't Be Replicated, Leading to Serious Questions about What's True and False in the World of Research

Is there a Cheshire Cat in science? One might believe so, given the many published scientific discoveries that cannot be independently reproduced. The ?replication crisis? in science has become a widely discussed issue among scientists and the lay media and even has its own entry in Wikipedia.

Statistical Considerations in Environmental Microbial Forensics

In environmental microbial forensics, as in other pursuits, statistical calculations are sometimes inappropriately applied, giving rise to the appearance of support for a particular conclusion or failing to support an innately obvious conclusion. This is a reflection of issues related to dealing with sample sizes, the methodologies involved, and the difficulty of communicating uncertainties. In this brief review, we attempt to illustrate ways to minimize such problems. In doing so, we consider one of the most common applications of environmental microbial forensics-the use of genotyping in food and water and disease investigations. We explore three important questions. (i) Do hypothesis tests' P values serve as adequate metrics of evidence? (ii) How can we quantify the value of the evidence? (iii) Can we turn a value-of-evidence metric into attribution probabilities? Our general conclusions are as follows. (i) P values have the unfortunate property of regularly detecting trivial effects when sample sizes are large. (ii) Likelihood ratios, rather than any kind of probability, are the better strength-of-evidence metric, addressing the question "what do these data say?" (iii) Attribution probabilities, addressing the question "what should I believe?," can be calculated using Bayesian methods, relying in part on likelihood ratios but also invoking prior beliefs which therefore can be quite subjective. In legal settings a Bayesian analysis may be required, but the choice and sensitivity of prior assumptions should be made clear.

Transparency in Ecology and Evolution: Real Problems, Real Solutions

To make progress scientists need to know what other researchers have found and how they found it. However, transparency is often insufficient across much of ecology and evolution. Researchers often fail to report results and methods in detail sufficient to permit interpretation and meta-analysis, and many results go entirely unreported. Further, these unreported results are often a biased subset. Thus the conclusions we can draw from the published literature are themselves often biased and sometimes might be entirely incorrect. Fortunately there is a movement across empirical disciplines, and now within ecology and evolution, to shape editorial policies to better promote transparency. This can be done by either requiring more disclosure by scientists or by developing incentives to encourage disclosure.