Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

On the Efficacy of Accuracy Prompts Across Partisan Lines: An Adversarial Collaboration

The spread of misinformation is a pressing societal challenge. Prior work shows that shifting attention to accuracy increases the quality of people's news-sharing decisions. However, researchers disagree on whether accuracy-prompt interventions work for U.S. Republicans/conservatives and whether partisanship moderates the effect. In this preregistered adversarial collaboration, we tested this question using a multiverse meta-analysis (k = 21; N = 27,828). In all 70 models, accuracy prompts improved sharing discernment among Republicans/conservatives. We observed significant partisan moderation for single-headline "evaluation" treatments (a critical test for one research team) such that the effect was stronger among Democrats than Republicans. However, this moderation was not consistently robust across different operationalizations of ideology/partisanship, exclusion criteria, or treatment type. Overall, we observed significant partisan moderation in 50% of specifications (all of which were considered critical for the other team). We discuss the conditions under which moderation is observed and offer interpretations.

Unraveling the genetics of arsenic toxicity with cellular morphology QTL

The health risks that arise from environmental exposures vary widely within and across human populations, and these differences are largely determined by genetic variation and gene-by-environment (gene-environment) interactions. However, risk assessment in laboratory mice typically involves isogenic strains and therefore, does not account for these known genetic effects. In this context, genetically heterogenous cell lines from laboratory mice are promising tools for population-based screening because they provide a way to introduce genetic variation in risk assessment without increasing animal use. Cell lines from genetic reference populations of laboratory mice offer genetic diversity, power for genetic mapping, and potentially, predictive value for in vivo experimentation in genetically matched individuals. To explore this further, we derived a panel of fibroblast lines from a genetic reference population of laboratory mice (the Diversity Outbred, DO). We then used high-content imaging to capture hundreds of cell morphology traits in cells exposed to the oxidative stress-inducing arsenic metabolite monomethylarsonous acid (MMAIII). We employed dose-response modeling to capture latent parameters of response and we then used these parameters to identify several hundred cell morphology quantitative trait loci (cmQTL). Response cmQTL encompass genes with established associations with cellular responses to arsenic exposure, including Abcc4 and Txnrd1, as well as novel gene candidates like Xrcc2. Moreover, baseline trait cmQTL highlight the influence of natural variation on fundamental aspects of nuclear morphology. We show that the natural variants influencing response include both coding and non-coding variation, and that cmQTL haplotypes can be used to predict response in orthogonal cell lines. Our study sheds light on the major molecular initiating events of oxidative stress that are under genetic regulation, including the NRF2-mediated antioxidant response, cellular detoxification pathways, DNA damage repair response, and cell death trajectories.

Grilling the data: application of specification curve analysis to red meat and all-cause mortality

OBJECTIVES

To present an application of specification curve analysis-a novel analytic method that involves defining and implementing all plausible and valid analytic approaches for addressing a research question-to nutritional epidemiology.

STUDY DESIGN AND SETTING

We reviewed all observational studies addressing the effect of red meat on all-cause mortality, sourced from a published systematic review, and documented variations in analytic methods (eg, choice of model, covariates, etc.). We enumerated all defensible combinations of analytic choices to produce a comprehensive list of all the ways in which the data may reasonably be analyzed. We applied specification curve analysis to data from National Health and Nutrition Examination Survey 2007 to 2014 to investigate the effect of unprocessed red meat on all-cause mortality. The specification curve analysis used a random sample of all reasonable analytic specifications we sourced from primary studies.

RESULTS

Among 15 publications reporting on 24 cohorts included in the systematic review on red meat and all-cause mortality, we identified 70 unique analytic methods, each including different analytic models, covariates, and operationalizations of red meat (eg, continuous vs quantiles). We applied specification curve analysis to National Health and Nutrition Examination Survey, including 10,661 participants. Our specification curve analysis included 1208 unique analytic specifications, of which 435 (36.0%) yielded a hazard ratio equal to or more than 1 for the effect of red meat on all-cause mortality and 773 (64.0%) less than 1. The specification curve analysis yielded a median hazard ratio of 0.94 (interquartile range: 0.83-1.05). Forty-eight specifications (3.97%) were statistically significant, 40 of which indicated unprocessed red meat to reduce all-cause mortality and eight of which indicated red meat to increase mortality.

CONCLUSION

We show that the application of specification curve analysis to nutritional epidemiology is feasible and presents an innovative solution to analytic flexibility.

Artificial intelligence and illusions of understanding in scientific research

Scientists are enthusiastically imagining ways in which artificial intelligence (AI) tools might improve research. Why are AI tools so attractive and what are the risks of implementing them across the research pipeline? Here we develop a taxonomy of scientists' visions for AI, observing that their appeal comes from promises to improve productivity and objectivity by overcoming human shortcomings. But proposed AI solutions can also exploit our cognitive limitations, making us vulnerable to illusions of understanding in which we believe we understand more about the world than we actually do. Such illusions obscure the scientific community's ability to see the formation of scientific monocultures, in which some types of methods, questions and viewpoints come to dominate alternative approaches, making science less innovative and more vulnerable to errors. The proliferation of AI tools in science risks introducing a phase of scientific enquiry in which we produce more but understand less. By analysing the appeal of these tools, we provide a framework for advancing discussions of responsible knowledge production in the age of AI.

Mediational roles of stress-coping factors in the relationship between patient-perceived communication quality and physical functioning: racial difference between Chinese and Non-Hispanic White American breast cancer survivors

PURPOSE

The assumption that patient-provider communication may mediate patients' sense of control over cancer to affect health outcomes has limited evidence. This study examines whether patient-perceived cancer care communication quality (PPCQ) mediates stress appraisal and coping behavior, affecting physical functioning across different racial groups.

METHODS

Two hundred and twenty Chinese American and 216 non-Hispanic White (NHW) women (ages 28-80) with stage 0-III breast cancer, 1-5 years post-diagnosis, and without recurrence, enrolled and completed a cross-sectional telephone survey. Physical functioning was measured by the NIH-PROMIS short form. Validated measures of PPCQ, patients' evaluation of their socioeconomic well-being, stress appraisal (perceived severity and control), use of coping strategies, treatment-related symptoms, and comorbidities were also assessed. Path analyses were used to examine the mediation for each racial group.

RESULTS

Regardless of race, treatment-related symptoms, comorbidities, and socioeconomic well-being were all directly related to physical functioning (p < 0.05). The impact of PPCQ on physical functioning was mediated by perceived control in the Chinese American group (p < 0.05), but not in the NHW group. Perceived severity and coping were not mediators of physical functioning in either group.

CONCLUSIONS

The mediational pathway from PPCQ to perceived control to physical functioning in Chinese American survivors may be partially explained by their lower socioeconomic well-being and culturally valued conformity to physicians as a medical authority. These sociocultural dynamics reinforce the importance of cancer care communication. For NHW survivors, the impact of treatment-related symptoms and socioeconomic well-being on physical functioning outweighed their PPCQ and perceived control.

The positive-negative-competence (PNC) model of psychological responses to representations of robots

Robots are becoming an increasingly prominent part of society. Despite their growing importance, there exists no overarching model that synthesizes people's psychological reactions to robots and identifies what factors shape them. To address this, we created a taxonomy of affective, cognitive and behavioural processes in response to a comprehensive stimulus sample depicting robots from 28 domains of human activity (for example, education, hospitality and industry) and examined its individual difference predictors. Across seven studies that tested 9,274 UK and US participants recruited via online panels, we used a data-driven approach combining qualitative and quantitative techniques to develop the positive-negative-competence model, which categorizes all psychological processes in response to the stimulus sample into three dimensions: positive, negative and competence-related. We also established the main individual difference predictors of these dimensions and examined the mechanisms for each predictor. Overall, this research provides an in-depth understanding of psychological functioning regarding representations of robots.

Care to share? Experimental evidence on code sharing behavior in the social sciences

Transparency and peer control are cornerstones of good scientific practice and entail the replication and reproduction of findings. The feasibility of replications, however, hinges on the premise that original researchers make their data and research code publicly available. This applies in particular to large-N observational studies, where analysis code is complex and may involve several ambiguous analytical decisions. To investigate which specific factors influence researchers' code sharing behavior upon request, we emailed code requests to 1,206 authors who published research articles based on data from the European Social Survey between 2015 and 2020. In this preregistered multifactorial field experiment, we randomly varied three aspects of our code request's wording in a 2x4x2 factorial design: the overall framing of our request (enhancement of social science research, response to replication crisis), the appeal why researchers should share their code (FAIR principles, academic altruism, prospect of citation, no information), and the perceived effort associated with code sharing (no code cleaning required, no information). Overall, 37.5% of successfully contacted authors supplied their analysis code. Of our experimental treatments, only framing affected researchers' code sharing behavior, though in the opposite direction we expected: Scientists who received the negative wording alluding to the replication crisis were more likely to share their research code. Taken together, our results highlight that the availability of research code will hardly be enhanced by small-scale individual interventions but instead requires large-scale institutional norms.

On the scope of scientific hypotheses

Hypotheses are frequently the starting point when undertaking the empirical portion of the scientific process. They state something that the scientific process will attempt to evaluate, corroborate, verify or falsify. Their purpose is to guide the types of data we collect, analyses we conduct, and inferences we would like to make. Over the last decade, metascience has advocated for hypotheses being in preregistrations or registered reports, but how to formulate these hypotheses has received less attention. Here, we argue that hypotheses can vary in specificity along at least three independent dimensions: the relationship, the variables, and the pipeline. Together, these dimensions form the scope of the hypothesis. We demonstrate how narrowing the scope of a hypothesis in any of these three ways reduces the hypothesis space and that this reduction is a type of novelty. Finally, we discuss how this formulation of hypotheses can guide researchers to formulate the appropriate scope for their hypotheses and should aim for neither too broad nor too narrow a scope. This framework can guide hypothesis-makers when formulating their hypotheses by helping clarify what is being tested, chaining results to previous known findings, and demarcating what is explicitly tested in the hypothesis.

Origin-country gender norms, individual work experience, and employment among immigrant women in Sweden

Many Western countries are built on a dual-earner model and have high levels of female labor force participation. Increasing the labor market activity of immigrant women is therefore seen as a key part of immigrant integration. However, female labor force participation (LFP) differs substantially between countries, reflecting differences in work-related gender norms that can continue to influence preferences and behaviors after migration. In this study, we investigate how origin-country gender norms and migrant selection interact to produce post-migration outcomes. Our data shows that immigrant women in Sweden have a higher level of pre-migration work experience than expected based on origin-country female LFP, indicating positive selection. Furthermore, the association between origin-country LFP and post-migration employment varied with work experience. For women without origin-country work experience, origin-country LFP was positively associated with employment in Sweden. For women with origin-country work experience, origin-country LFP however was not associated with higher likelihood of employment in Sweden. Though our focus is on immigrant women, we also include immigrant men in our analysis to test our prediction more thoroughly. For men without origin-country work experience, origin-country LFP was negatively associated with employment in Sweden, while we found no association for men with origin-country work experience. Our results show that migrant selection is a crucial factor in understanding the relationship between origin-country LFP and post-migration labor market outcomes, and that these patterns vary with gender. Policy interventions targeting immigrant women from countries with low female LFP should therefore not assume that women arrive socialized with gender-norms that hinder labor market activity.

One data set, many analysts: Implications for practicing scientists

Researchers routinely face choices throughout the data analysis process. It is often opaque to readers how these choices are made, how they affect the findings, and whether or not data analysis results are unduly influenced by subjective decisions. This concern is spurring numerous investigations into the variability of data analysis results. The findings demonstrate that different teams analyzing the same data may reach different conclusions. This is the "many-analysts" problem. Previous research on the many-analysts problem focused on demonstrating its existence, without identifying specific practices for solving it. We address this gap by identifying three pitfalls that have contributed to the variability observed in many-analysts publications and providing suggestions on how to avoid them.