Variability in the analysis of a single neuroimaging dataset by many teams

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

Third-Party Cooperation: How Reducing Material Involvement Enhances Contributions to the Public Good

Decisions to cooperate are often delegated to a third party. We examined whether cooperation differs when decisions are made for a third party compared with ourselves and specified which motives are important for third-party cooperation. Participants played multiple rounds of a public goods game (PGG). In Study 1, we varied personal involvement from high to low; participants played for themselves (Self), for themselves and a third party (Shared), and solely for a third party (Third Party). Participants contributed most when personal involvement was lowest (i.e., Third Party) and least when personal involvement was high (i.e., Self). Study 2 explored if social motives underlie third-party cooperation by comparing cooperation with social (human) and non-social (computer) group members. Reducing personal involvement in the PGG (i.e., Third Party) increased cooperation in social contexts compared with non-social contexts, indicating enhanced collective interest. Increased cooperation for a third party may result from taking the other's perspective, thereby increasing social norm preferences.

The Effect of Positive and Negative Feedback on Risk-Taking across Different Contexts

Preferences for risky choices have often been shown to be unstable and context-dependent. Though people generally avoid gambles with mixed outcomes, a phenomenon often attributed to loss aversion, contextual factors can impact this dramatically. For example, people typically prefer risky options after a financial loss, while generally choosing safer options after a monetary gain. However, it is unclear what exactly contributes to these preference shifts as a function of prior outcomes, as these gain/loss outcomes are usually confounded with participant performance, and therefore it is unclear whether these effects are driven purely by the monetary gains or losses, or rather by success or failure at the actual task. Here, we experimentally separated the effects of monetary gains/losses from performance success/failure prior to a standard risky choice. Participants performed a task in which they experienced contextual effects: 1) monetary gain or loss based directly on performance, 2) monetary gain or loss that was randomly awarded and was, crucially, independent from performance, and 3) success or failure feedback based on performance, but without any monetary incentive. Immediately following these positive/negative contexts, participants were presented with a gain-loss gamble that they had to decide to either play or pass. We found that risk preferences for identical sets of gambles were biased by positive and negative contexts containing monetary gains and losses, but not by contexts containing performance feedback. This data suggests that the observed framing effects are driven by aversion for monetary losses and not simply by the positive or negative valence of the context, or by potential moods resulting from positive or negative contexts. These results highlight the specific context dependence of risk preferences.