Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 ( Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.

A new dual-purpose ultrahigh vacuum infrared spectroscopy apparatus optimized for grazing-incidence reflection as well as for transmission geometries

A newly designed ultrahigh vacuum (UHV) infrared spectroscopy apparatus dedicated to the spectroscopic characterization of oxides, singles crystals as well as powders, is described. It combines a state-of-the-art vacuum Fourier transform infrared (FTIR) spectrometer (Bruker, VERTEX 80v) with a novel UHV system (PREVAC) consisting of load-lock, distribution, measurement, and magazine chambers. The innovative design allows carrying out both reflection-absorption IR spectroscopy experiments at grazing incidence on well-defined oxide single crystal surfaces and FTIR transmission measurements for powder particles. A further unique feature of the apparatus is the entirely evacuated optical path to avoid background signals from gas phase H(2)O, CO(2), and other species, thus creating the possibility to record high-quality IR data with high sensitivity and stability, an essential prerequisite for monitoring molecular species adsorbed on oxide single-crystal surfaces. The unique performance of this new apparatus with regard to the spectroscopic characterization of adsorbates on oxide single crystals as well as on powder particles is demonstrated by case studies for two different materials, TiO(2) and ZnO.