1
|
Auspurg K, Brüderl J. Toward a more credible assessment of the credibility of science by many-analyst studies. Proc Natl Acad Sci U S A 2024; 121:e2404035121. [PMID: 39236231 PMCID: PMC11420151 DOI: 10.1073/pnas.2404035121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/07/2024] Open
Abstract
We discuss a relatively new meta-scientific research design: many-analyst studies that attempt to assess the replicability and credibility of research based on large-scale observational data. In these studies, a large number of analysts try to answer the same research question using the same data. The key idea is the greater the variation in results, the greater the uncertainty in answering the research question and, accordingly, the lower the credibility of any individual research finding. Compared to individual replications, the large crowd of analysts allows for a more systematic investigation of uncertainty and its sources. However, many-analyst studies are also resource-intensive, and there are some doubts about their potential to provide credible assessments. We identify three issues that any many-analyst study must address: 1) identifying the source of variation in the results; 2) providing an incentive structure similar to that of standard research; and 3) conducting a proper meta-analysis of the results. We argue that some recent many-analyst studies have failed to address these issues satisfactorily and have therefore provided an overly pessimistic assessment of the credibility of science. We also provide some concrete guidance on how future many-analyst studies could provide a more constructive assessment.
Collapse
Affiliation(s)
- Katrin Auspurg
- Department of Sociology, Ludwig-Maximilians-Universität (LMU) Munich, Munich80801, Germany
| | - Josef Brüderl
- Department of Sociology, Ludwig-Maximilians-Universität (LMU) Munich, Munich80801, Germany
| |
Collapse
|
2
|
Hoogeveen S, Borsboom D, Kucharský Š, Marsman M, Molenaar D, de Ron J, Sekulovski N, Visser I, van Elk M, Wagenmakers EJ. Prevalence, patterns and predictors of paranormal beliefs in The Netherlands: a several-analysts approach. ROYAL SOCIETY OPEN SCIENCE 2024; 11:240049. [PMID: 39233722 PMCID: PMC11371428 DOI: 10.1098/rsos.240049] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/09/2024] [Revised: 03/20/2024] [Accepted: 03/21/2024] [Indexed: 09/06/2024]
Abstract
Paranormal beliefs encompass a wide variety of phenomena, including the existence of supernatural entities such as ghosts and witches, as well as extraordinary human abilities such as telepathy and clairvoyance. In the current study, we used a nationally representative sample ( N = 2534 ) to investigate the presence and correlates of paranormal beliefs among the secular Dutch population. The results indicated that most single paranormal phenomena (e.g. belief in clairvoyance) are endorsed by 10-20% of Dutch respondents; however, 55.6% of respondents qualify as paranormal believers based on the preregistered criterion that they believe in at least one phenomenon with considerable certainty. In addition, we invited four analysis teams with different methodological expertise to assess the structure of paranormal beliefs using traditional factor analysis, network analysis, Bayesian network analysis and latent class analysis (LCA). The teams' analyses indicated adequate fit of a four-factor structure reported in a 1985 study, but also emphasized different conclusions across techniques; network analyses showed evidence against strong connectedness within most clusters, and suggested a five-cluster structure. The application of various analytic techniques painted a nuanced picture of paranormal beliefs and believers in The Netherlands and suggests that despite increased secularization, subgroups of the general population still believe in paranormal phenomena.
Collapse
Affiliation(s)
- S. Hoogeveen
- Department of Methodology and Statistics, Utrecht University, Utrecht, The Netherlands
| | - D. Borsboom
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | - Š. Kucharský
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | - M. Marsman
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | - D. Molenaar
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | - J. de Ron
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | - N. Sekulovski
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | - I. Visser
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | - M. van Elk
- Institute of Psychology, Leiden University, Leiden, The Netherlands
| | - E.-J. Wagenmakers
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
| |
Collapse
|
3
|
Holzmeister F, Johannesson M, Böhm R, Dreber A, Huber J, Kirchler M. Heterogeneity in effect size estimates. Proc Natl Acad Sci U S A 2024; 121:e2403490121. [PMID: 39078672 PMCID: PMC11317577 DOI: 10.1073/pnas.2403490121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/19/2024] [Accepted: 06/28/2024] [Indexed: 07/31/2024] Open
Abstract
A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.
Collapse
Affiliation(s)
- Felix Holzmeister
- Department of Economics, University of Innsbruck, A-6020Innsbruck, Austria
| | - Magnus Johannesson
- Department of Economics, Stockholm School of Economics, SE-113 83Stockholm, Sweden
| | - Robert Böhm
- Department of Occupational, Economic, and Social Psychology, University of Vienna, A-1010Vienna, Austria
- Department of Psychology and Center for Social Data Science, University of Copenhagen, DK-1353Copenhagen, Denmark
| | - Anna Dreber
- Department of Economics, University of Innsbruck, A-6020Innsbruck, Austria
- Department of Economics, Stockholm School of Economics, SE-113 83Stockholm, Sweden
| | - Jürgen Huber
- Department of Banking and Finance, University of Innsbruck, A-6020Innsbruck, Austria
| | - Michael Kirchler
- Department of Banking and Finance, University of Innsbruck, A-6020Innsbruck, Austria
| |
Collapse
|
4
|
Götz M, Sarma A, O'Boyle EH. The multiverse of universes: A tutorial to plan, execute and interpret multiverses analyses using the R package multiverse. INTERNATIONAL JOURNAL OF PSYCHOLOGY 2024. [PMID: 39030767 DOI: 10.1002/ijop.13229] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2024] [Accepted: 06/27/2024] [Indexed: 07/22/2024]
Abstract
Even when guided by strong theories and sound methods, researchers must often choose a singular course of action from multiple viable alternatives. Regardless of the choice, it, along with all other choices made during the research process, individually and collectively affects study results, often in unpredictable ways. The inability to disentangle how much of an observed effect is attributable to the phenomenon of interest, and how much is attributable to what have come to be known as researcher degrees of freedom (RDF), slows theoretical progress and stymies practical implementation. However, if one could examine the results from a particular set of RDF (known as a universe) against a systematically and comprehensively determined background of alternative viable universes (known as a multiverse), then the effects of RDF can be directly examined to provide greater context and clarity to future researchers, and greater confidence in the recommendations to practitioners. This tutorial demonstrates a means to map result variability directly and efficiently, and empirically investigate RDF impact on conclusions via multiverse analysis. Using the R package multiverse, we outline best practices in planning, executing and interpreting of multiverse analyses.
Collapse
|
5
|
Sarafoglou A, Hoogeveen S, van den Bergh D, Aczel B, Albers CJ, Althoff T, Botvinik-Nezer R, Busch NA, Cataldo AM, Devezer B, van Dongen NNN, Dreber A, Fried EI, Hoekstra R, Hoffman S, Holzmeister F, Huber J, Huntington-Klein N, Ioannidis J, Johannesson M, Kirchler M, Loken E, Mangin JF, Matzke D, Menkveld AJ, Nilsonne G, van Ravenzwaaij D, Schweinsberg M, Schulz-Kuempel H, Shanks DR, Simons DJ, Spellman BA, Stoevenbelt AH, Szaszi B, Trübutschek D, Tuerlinckx F, Uhlmann EL, Vanpaemel W, Wicherts J, Wagenmakers EJ. Subjective evidence evaluation survey for many-analysts studies. ROYAL SOCIETY OPEN SCIENCE 2024; 11:240125. [PMID: 39050728 PMCID: PMC11265885 DOI: 10.1098/rsos.240125] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/01/2024] [Accepted: 04/22/2024] [Indexed: 07/27/2024]
Abstract
Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.
Collapse
Affiliation(s)
| | | | - Don van den Bergh
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | - Balazs Aczel
- Institute of Psychology, ELTE Eötvös Lorénd University, Budapest, Hungary
| | - Casper J. Albers
- Heymans Institute for Psychological Research, University of Groningen, Groningen, The Netherlands
| | - Tim Althoff
- Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA
| | - Rotem Botvinik-Nezer
- Hebrew University of Jerusalem, Jerusalem, Israel
- Dartmouth College, Hanover, NH, USA
| | - Niko A. Busch
- Institute for Psychology, University of Münster, Münster, Germany
| | - Andrea M. Cataldo
- Center for Depression, Anxiety and Stress Research, McLean Hospital, Belmont, MA, USA
- Department of Psychiatry, Harvard Medical School, Boston, MA, USA
| | - Berna Devezer
- Department of Business, University of Idaho, Moscow, ID, USA
| | | | - Anna Dreber
- Stockholm School of Economics, Stockholm, Sweden
- University of Innsbruck, Innsbruck, Tirol, Austria
| | - Eiko I. Fried
- Department of Psychology, Leiden University, Leiden, The Netherlands
| | - Rink Hoekstra
- Nieuwenhuis Institute for Educational Research, University of Groningen, Groningen, The Netherlands
| | - Sabine Hoffman
- Department of Statistics, Ludwig-Maximilians-Universität München, Munchen, Bayern, Germany
| | | | - Jürgen Huber
- University of Innsbruck, Innsbruck, Tirol, Austria
| | | | - John Ioannidis
- Meta-Research Innovation Center at Stanford (METRICS) and Departments of Medicine, of Epidemiology and of Population Health, of Biomedical Data Science, and of Statistics, Stanford University, Stanford, CA, USA
| | | | | | - Eric Loken
- University of Conneticut, Storrs, CT, USA
| | - Jan-Francois Mangin
- University Paris-Saclay, Gif-sur-Yvette, France
- Neurospin CEA, Gif-sur-Yvette, Île-de-France, France
| | - Dora Matzke
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | | | | | - Don van Ravenzwaaij
- Heymans Institute for Psychological Research, University of Groningen, Groningen, The Netherlands
| | | | - Hannah Schulz-Kuempel
- Department of Statistics and The Institute for Medical Information Processing, Biometry, and Epidemiology, LMU Munich, Munchen, Bayern, Germany
- The Institute for Medical Information Processing, Biometry, and Epidemiology, LMU Munich, Munchen, Bayern, Germany
| | - David R. Shanks
- Division of Psychology and Language Sciences, University College London, 26 Bedford Way, London WC1H 0AP, UK
| | | | - Barbara A. Spellman
- School of Law, University of Virginia, 580 Massie Road, Charlottesville, VA, USA
| | - Andrea H. Stoevenbelt
- Nieuwenhuis Institute for Educational Research, University of Groningen, Groningen, The Netherlands
| | - Barnabas Szaszi
- Institute of Psychology, ELTE Eötvös Lorénd University, Budapest, Hungary
| | | | | | | | | | - Jelte Wicherts
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| | | |
Collapse
|
6
|
Mukherjee S, Mattos RS, Toldo JM, Lischka H, Barbatti M. Prediction Challenge: Simulating Rydberg photoexcited cyclobutanone with surface hopping dynamics based on different electronic structure methods. J Chem Phys 2024; 160:154306. [PMID: 38624122 DOI: 10.1063/5.0203636] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2024] [Accepted: 03/28/2024] [Indexed: 04/17/2024] Open
Abstract
This research examines the nonadiabatic dynamics of cyclobutanone after excitation into the n → 3s Rydberg S2 state. It stems from our contribution to the Special Topic of the Journal of Chemical Physics to test the predictive capability of computational chemistry against unseen experimental data. Decoherence-corrected fewest-switches surface hopping was used to simulate nonadiabatic dynamics with full and approximated nonadiabatic couplings. Several simulation sets were computed with different electronic structure methods, including a multiconfigurational wavefunction [multiconfigurational self-consistent field (MCSCF)] specially built to describe dissociative channels, multireference semiempirical approach, time-dependent density functional theory, algebraic diagrammatic construction, and coupled cluster. MCSCF dynamics predicts a slow deactivation of the S2 state (10 ps), followed by an ultrafast population transfer from S1 to S0 (<100 fs). CO elimination (C3 channel) dominates over C2H4 formation (C2 channel). These findings radically differ from the other methods, which predicted S2 lifetimes 10-250 times shorter and C2 channel predominance. These results suggest that routine electronic structure methods may hold low predictive power for the outcome of nonadiabatic dynamics.
Collapse
Affiliation(s)
| | - Rafael S Mattos
- Aix Marseille University, CNRS, ICR, Marseille 13397, France
| | - Josene M Toldo
- Aix Marseille University, CNRS, ICR, Marseille 13397, France
| | - Hans Lischka
- Department of Chemistry and Biochemistry, Texas Tech University, Lubbock, Texas 79409-1061, USA
| | - Mario Barbatti
- Aix Marseille University, CNRS, ICR, Marseille 13397, France
- Institut Universitaire de France, Paris 75231, France
| |
Collapse
|
7
|
Carpentras D. We urgently need a culture of multi-operationalization in psychological research. COMMUNICATIONS PSYCHOLOGY 2024; 2:32. [PMID: 39242896 PMCID: PMC11332105 DOI: 10.1038/s44271-024-00084-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/22/2023] [Accepted: 03/25/2024] [Indexed: 09/09/2024]
Abstract
Analysis of different operationalizations shows that many scientific results may be an artifact of the operationalization process. A culture of multi-operationalization may be needed for psychological research to develop valid knowledge.
Collapse
Affiliation(s)
- Dino Carpentras
- ETH Zürich, Computational Social Science, Stampfenbachstrasse 48, 8092, Zürich, Switzerland.
| |
Collapse
|
8
|
Nosrat C, Martin-Tuite P, Jiang F, Broering J, Shindel AW. Gender Bias in Letters of Recommendation: Relevance to Urology Match Outcomes and Pursuit of Fellowship Training/Academic Career. Urology 2024; 183:281-287. [PMID: 37940078 DOI: 10.1016/j.urology.2023.09.053] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2023] [Revised: 09/02/2023] [Accepted: 09/06/2023] [Indexed: 11/10/2023]
Abstract
OBJECTIVE To review applications to a single urology residency program to determine application characteristics predictive of (1) successful match into urology residency and (2) pursuit of fellowship training and/or academic practice after completion of residency. Our principal variables of interest were gender bias as assessed in letters of recommendation (LOR), personal statements, Medical Student Performance Evaluation (MSPE), race, and gender. MATERIALS AND METHODS Applications submitted to our urology residency program in the 2014 cycle were reviewed. Twenty-three variables were analyzed, including applicant demographics, application materials, and gender bias. Deidentified text from LOR, personal statements, and MSPE was evaluated for gender bias using an open-source gender bias calculator. A subanalysis of applicants who matched at a top 25 urology program was performed. Logistic regression analysis was performed to identify applicant variables associated with (1) match success and (2) fellowship training or academic employment as of September 2021. RESULTS Two hundred and twenty-two completed applications were analyzed. First authorship of a published manuscript was significantly associated with greater odds of matching. Female gender and top 25 medical school attendance were both significant predictors of matching at a top 25 urology program. The number of first-author publications was associated with completion of fellowship training or current employment in an academic position. CONCLUSION First-author publications are the most important preinterview determinant of match success and subsequent pursuit of academic practice/fellowship training. Certain applicant characteristics are associated with matching at highly ranked programs. Gender bias in application materials (including LOR) does not appear to exert a significant influence on match and early career outcomes.
Collapse
Affiliation(s)
- Cameron Nosrat
- School of Medicine, University of California-San Francisco, San Francisco, CA.
| | - Patrick Martin-Tuite
- Department of Surgery, Division of Urology, Washington University in Saint Louis, St. Louis, MO
| | - Fei Jiang
- Department of Epidemiology and Biostatistics, University of California-San Francisco, San Francisco, CA
| | - Jenny Broering
- Department of Urology, University of California-San Francisco, San Francisco, CA
| | - Alan W Shindel
- Department of Urology, University of California-San Francisco, San Francisco, CA
| |
Collapse
|
9
|
Cole VT, Hussong AM, Gottfredson NC, Bauer DJ, Curran PJ. Informing Harmonization Decisions in Integrative Data Analysis: Exploring the Measurement Multiverse. PREVENTION SCIENCE : THE OFFICIAL JOURNAL OF THE SOCIETY FOR PREVENTION RESEARCH 2023; 24:1595-1607. [PMID: 36441362 DOI: 10.1007/s11121-022-01466-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/27/2022] [Indexed: 11/29/2022]
Abstract
Combining datasets in an integrative data analysis (IDA) requires researchers to make a number of decisions about how best to harmonize item responses across datasets. This entails two sets of steps: logical harmonization, which involves combining items which appear similar across datasets, and analytic harmonization, which involves using psychometric models to find and account for cross-study differences in measurement. Embedded in logical and analytic harmonization are many decisions, from deciding whether items can be combined prima facie to how best to find covariate effects on specific items. Researchers may not have specific hypotheses about these decisions, and each individual choice may seem arbitrary, but the cumulative effects of these decisions are unknown. In the current study, we conducted an IDA of the relationship between alcohol use and delinquency using three datasets (total N = 2245). For analytic harmonization, we used moderated nonlinear factor analysis (MNLFA) to generate factor scores for delinquency. We conducted both logical and analytic harmonization 72 times, each time making a different set of decisions. We assessed the cumulative influence of these decisions on MNLFA parameter estimates, factor scores, and estimates of the relationship between delinquency and alcohol use. There were differences across paths in MNLFA parameter estimates, but fewer differences in estimates of factor scores and regression parameters linking delinquency to alcohol use. These results suggest that factor scores may be relatively robust to subtly different decisions in data harmonization, and measurement model parameters are less so.
Collapse
Affiliation(s)
- Veronica T Cole
- Department of Psychology, Wake Forest University, 1834 Wake Forest Road, Winston-Salem, NC, 27109, USA.
| | - Andrea M Hussong
- Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Nisha C Gottfredson
- Department of Health Behavior, Gillings School of Global Public Health, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Daniel J Bauer
- Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Patrick J Curran
- Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| |
Collapse
|
10
|
Sirois S, Brisson J, Blaser E, Calignano G, Donenfeld J, Hepach R, Hochmann JR, Kaldy Z, Liszkowski U, Mayer M, Ross-Sheehy S, Russo S, Valenza E. The pupil collaboration: A multi-lab, multi-method analysis of goal attribution in infants. Infant Behav Dev 2023; 73:101890. [PMID: 37944367 DOI: 10.1016/j.infbeh.2023.101890] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2022] [Revised: 09/27/2023] [Accepted: 09/27/2023] [Indexed: 11/12/2023]
Abstract
The rise of pupillometry in infant research over the last decade is associated with a variety of methods for data preprocessing and analysis. Although pupil diameter is increasingly recognized as an alternative measure of the popular cumulative looking time approach used in many studies (Jackson & Sirois, 2022), an open question is whether the many approaches used to analyse this variable converge. To this end, we proposed a crowdsourced approach to pupillometry analysis. A dataset from 30 9-month-old infants (15 girls; Mage = 282.9 days, SD = 8.10) was provided to 7 distinct teams for analysis. The data were obtained from infants watching video sequences showing a hand, initially resting between two toys, grabbing one of them (after Woodward, 1998). After habituation, infants were shown (in random order) a sequence of four test events that varied target position and target toy. Results show that looking times reflect primarily the familiar path of the hand, regardless of target toy. Gaze data similarly show this familiarity effect of path. The pupil dilation analyses show that features of pupil baseline measures (duration and temporal location) as well as data retention variation (trial and/or participant) due to different inclusion criteria from the various analysis methods are linked to divergences in findings. Two of the seven teams found no significant findings, whereas the remaining five teams differ in the pattern of findings for main and interaction effects. The discussion proposes guidelines for best practice in the analysis of pupillometry data.
Collapse
Affiliation(s)
- Sylvain Sirois
- Département de Psychologie, Université du Québec à Trois-Rivières, Canada.
| | - Julie Brisson
- Centre de Recherche sur les fonctionnements et dysfonctionnements psychologiques (EA7475), Université de Rouen Normandie, France
| | - Erik Blaser
- Department of Psychology, University of Massachusetts Boston, USA
| | - Giulia Calignano
- Department of Developmental and Social Psychology, University of Padova, Italy
| | - Jamie Donenfeld
- Department of Psychology, University of Massachusetts Boston, USA
| | - Robert Hepach
- Department of Experimental Psychology, University of Oxford, UK
| | - Jean-Rémy Hochmann
- CNRS UMR5229 - Institut des Sciences Cognitives Marc Jeannerod, Université Lyon 1, France
| | - Zsuzsa Kaldy
- Department of Psychology, University of Massachusetts Boston, USA
| | - Ulf Liszkowski
- Department of Developmental Psychology, University of Hamburg, Germany
| | - Marlena Mayer
- Department of Developmental Psychology, University of Hamburg, Germany
| | | | - Sofia Russo
- Department of Developmental and Social Psychology, University of Padova, Italy
| | - Eloisa Valenza
- Department of Developmental and Social Psychology, University of Padova, Italy
| |
Collapse
|
11
|
Krpan D, Booth JE, Damien A. The positive-negative-competence (PNC) model of psychological responses to representations of robots. Nat Hum Behav 2023; 7:1933-1954. [PMID: 37783891 PMCID: PMC10663151 DOI: 10.1038/s41562-023-01705-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2022] [Accepted: 08/25/2023] [Indexed: 10/04/2023]
Abstract
Robots are becoming an increasingly prominent part of society. Despite their growing importance, there exists no overarching model that synthesizes people's psychological reactions to robots and identifies what factors shape them. To address this, we created a taxonomy of affective, cognitive and behavioural processes in response to a comprehensive stimulus sample depicting robots from 28 domains of human activity (for example, education, hospitality and industry) and examined its individual difference predictors. Across seven studies that tested 9,274 UK and US participants recruited via online panels, we used a data-driven approach combining qualitative and quantitative techniques to develop the positive-negative-competence model, which categorizes all psychological processes in response to the stimulus sample into three dimensions: positive, negative and competence-related. We also established the main individual difference predictors of these dimensions and examined the mechanisms for each predictor. Overall, this research provides an in-depth understanding of psychological functioning regarding representations of robots.
Collapse
Affiliation(s)
- Dario Krpan
- Department of Psychological and Behavioural Science, London School of Economics and Political Science, London, UK.
| | - Jonathan E Booth
- Department of Management, London School of Economics and Political Science, London, UK
| | - Andreea Damien
- Department of Psychological and Behavioural Science, London School of Economics and Political Science, London, UK
| |
Collapse
|
12
|
Schweinsberg M, Thau S, Pillutla M. Research-Problem Validity in Primary Research: Precision and Transparency in Characterizing Past Knowledge. PERSPECTIVES ON PSYCHOLOGICAL SCIENCE 2023; 18:1230-1243. [PMID: 36745743 PMCID: PMC10475212 DOI: 10.1177/17456916221144990] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023]
Abstract
Four validity types evaluate the approximate truth of inferences communicated by primary research. However, current validity frameworks ignore the truthfulness of empirical inferences that are central to research-problem statements. Problem statements contrast a review of past research with other knowledge that extends, contradicts, or calls into question specific features of past research. Authors communicate empirical inferences, or quantitative judgments, about the frequency (e.g., "few," "most") and variability (e.g., "on the one hand," "on the other hand") in their reviews of existing theories, measures, samples, or results. We code a random sample of primary research articles and show that 83% of quantitative judgments in our sample are vague and do not have a transparent origin, making it difficult to assess their validity. We review validity threats of current practices. We propose that documenting the literature search, reporting how the search was coded, and quantifying the search results facilitates more precise judgments and makes their origin transparent. This practice enables research questions that are more closely tied to the existing body of knowledge and allows for more informed evaluations of the contribution of primary research articles, their design choices, and how they advance knowledge. We discuss potential limitations of our proposed framework.
Collapse
|
13
|
Keener SK, Kepes S, Torka AK. The trustworthiness of the cumulative knowledge in industrial/organizational psychology: The current state of affairs and a path forward. Acta Psychol (Amst) 2023; 239:104005. [PMID: 37625919 DOI: 10.1016/j.actpsy.2023.104005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2023] [Revised: 07/13/2023] [Accepted: 08/04/2023] [Indexed: 08/27/2023] Open
Abstract
The goal of industrial/organizational (IO) psychology, is to build and organize trustworthy knowledge about people-related phenomena in the workplace. Unfortunately, as with other scientific disciplines, our discipline may be experiencing a "crisis of confidence" stemming from the lack of reproducibility and replicability of many of our field's research findings, which would suggest that much of our research may be untrustworthy. If a scientific discipline's research is deemed untrustworthy, it can have dire consequences, including the withdraw of funding for future research. In this focal article, we review the current state of reproducibility and replicability in IO psychology and related fields. As part of this review, we discuss factors that make it less likely that research findings will be trustworthy, including the prevalence of scientific misconduct, questionable research practices (QRPs), and errors. We then identify some root causes of these issues and provide several potential remedies. In particular, we highlight the need for improved research methods and statistics training as well as a re-alignment of the incentive structure in academia. To accomplish this, we advocate for changes in the reward structure, improvements to the peer review process, and the implementation of open science practices. Overall, addressing the current "crisis of confidence" in IO psychology requires individual researchers, academic institutions, and publishers to embrace system-wide change.
Collapse
Affiliation(s)
- Sheila K Keener
- Department of Management, Old Dominion University, Norfolk, VA, United States of America.
| | - Sven Kepes
- Department of Management and Entrepreneurship, Virginia Commonwealth University, Richmond, VA, United States of America.
| | - Ann-Kathrin Torka
- Department of Social, Work, and Organizational Psychology, TU Dortmund University, Dortmund, Germany.
| |
Collapse
|
14
|
Botvinik-Nezer R, Wager TD. Reproducibility in Neuroimaging Analysis: Challenges and Solutions. BIOLOGICAL PSYCHIATRY. COGNITIVE NEUROSCIENCE AND NEUROIMAGING 2023; 8:780-788. [PMID: 36906444 DOI: 10.1016/j.bpsc.2022.12.006] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/15/2022] [Revised: 11/27/2022] [Accepted: 12/11/2022] [Indexed: 12/23/2022]
Abstract
Recent years have marked a renaissance in efforts to increase research reproducibility in psychology, neuroscience, and related fields. Reproducibility is the cornerstone of a solid foundation of fundamental research-one that will support new theories built on valid findings and technological innovation that works. The increased focus on reproducibility has made the barriers to it increasingly apparent, along with the development of new tools and practices to overcome these barriers. Here, we review challenges, solutions, and emerging best practices with a particular emphasis on neuroimaging studies. We distinguish 3 main types of reproducibility, discussing each in turn. Analytical reproducibility is the ability to reproduce findings using the same data and methods. Replicability is the ability to find an effect in new datasets, using the same or similar methods. Finally, robustness to analytical variability refers to the ability to identify a finding consistently across variation in methods. The incorporation of these tools and practices will result in more reproducible, replicable, and robust psychological and brain research and a stronger scientific foundation across fields of inquiry.
Collapse
Affiliation(s)
- Rotem Botvinik-Nezer
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, New Hampshire.
| | - Tor D Wager
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, New Hampshire
| |
Collapse
|
15
|
Lewis MW, Bradford DE, Pace-Schott EF, Rauch SL, Rosso IM. Multiverse analyses of fear acquisition and extinction retention in posttraumatic stress disorder. Psychophysiology 2023; 60:e14265. [PMID: 36786400 PMCID: PMC10330173 DOI: 10.1111/psyp.14265] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2022] [Revised: 12/13/2022] [Accepted: 01/14/2023] [Indexed: 02/15/2023]
Abstract
Persistent fear is a cardinal feature of posttraumatic stress disorder (PTSD), and deficient fear extinction retention is a proposed illness mechanism and target of exposure-based therapy. However, evidence for deficient fear extinction in PTSD has been mixed using laboratory paradigms, which may relate to underidentified methodological variation across studies. We reviewed the literature to identify parameters that differ across studies of fear extinction retention in PTSD. We then performed Multiverse Analysis in a new sample, to quantify the impact of those methodological parameters on statistical findings. In 25 PTSD patients (15 female) and 36 trauma-exposed non-PTSD controls (TENC) (20 female), we recorded skin conductance response (SCR) during fear acquisition and extinction learning (day 1) and extinction recall (day 2). A first Multiverse Analysis examined the effects of methodological parameters identified by the literature review on comparisons of SCR-based fear extinction retention in PTSD versus TENC. A second Multiverse Analysis examined the effects of those methodological parameters on comparisons of SCR to a danger cue (CS+) versus safety cue (CS-) during fear acquisition. Both the literature review and the Multiverse Analysis yielded inconsistent findings for fear extinction retention in PTSD versus TENC, and most analyses found no statistically significant group difference. By contrast, significantly elevated SCR to CS+ versus CS- was consistently found across all analyses in the literature review and the Multiverse Analysis of new data. We discuss methodological parameters that may most contribute to inconsistent findings of fear extinction retention deficit in PTSD and implications for future clinical research.
Collapse
Affiliation(s)
- Michael W. Lewis
- McLean Hospital, Center for Depression, Anxiety, and Stress Research
- Harvard Medical School, Department of Psychiatry
| | | | - Edward F. Pace-Schott
- Harvard Medical School, Department of Psychiatry
- Massachusetts General Hospital, Department of Psychiatry
| | - Scott L. Rauch
- McLean Hospital, Center for Depression, Anxiety, and Stress Research
- Harvard Medical School, Department of Psychiatry
| | - Isabelle M. Rosso
- McLean Hospital, Center for Depression, Anxiety, and Stress Research
- Harvard Medical School, Department of Psychiatry
| |
Collapse
|
16
|
Brown VA, Strand JF. Preregistration: Practical Considerations for Speech, Language, and Hearing Research. JOURNAL OF SPEECH, LANGUAGE, AND HEARING RESEARCH : JSLHR 2023; 66:1889-1898. [PMID: 36472937 PMCID: PMC10465155 DOI: 10.1044/2022_jslhr-22-00317] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/31/2022] [Revised: 07/18/2022] [Accepted: 08/27/2022] [Indexed: 06/17/2023]
Abstract
PURPOSE In the last decade, psychology and other sciences have implemented numerous reforms to improve the robustness of our research, many of which are based on increasing transparency throughout the research process. Among these reforms is the practice of preregistration, in which researchers create a time-stamped and uneditable document before data collection that describes the methods of the study, how the data will be analyzed, the sample size, and many other decisions. The current article highlights the benefits of preregistration with a focus on the specific issues that speech, language, and hearing researchers are likely to encounter, and additionally provides a tutorial for writing preregistrations. CONCLUSIONS Although rates of preregistration have increased dramatically in recent years, the practice is still relatively uncommon in research on speech, language, and hearing. Low rates of adoption may be driven by a lack of understanding of the benefits of preregistration (either generally or for our discipline in particular) or uncertainty about how to proceed if it becomes necessary to deviate from the preregistered plan. Alternatively, researchers may see the benefits of preregistration but not know where to start, and gathering this information from a wide variety of sources is arduous and time consuming. This tutorial addresses each of these potential roadblocks to preregistration and equips readers with tools to facilitate writing preregistrations for research on speech, language, and hearing. SUPPLEMENTAL MATERIAL https://doi.org/10.23641/asha.21644843.
Collapse
Affiliation(s)
- Violet A. Brown
- Department of Psychological & Brain Sciences, Washington University in St. Louis, MO
| | | |
Collapse
|
17
|
Haehner P, Bleidorn W, Hopwood CJ. Examining individual differences in personality trait changes after negative life events. EUROPEAN JOURNAL OF PERSONALITY 2023. [DOI: 10.1177/08902070231156840] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/18/2023]
Abstract
Personality traits can change throughout the entire life span, but people differ in their personality trait changes. To better understand individual differences in personality changes, we examined personal (personality functioning), environmental (environmental changes), and event-related moderators (e.g., perceived event characteristics) of personality trait changes. Therefore, we used a sample of 1069 participants who experienced a negative life event in the last 5 weeks and assessed their personality traits at five measurement occasions over 6 months. Employing preregistered multilevel lasso estimation, we did not find any significant effects. While exploratory analyses generally confirmed this conclusion, they also identified some effects that might being worth to be considered in future research (e.g., perceived impact and perceived social status changes were associated with changes in agreeableness after experiencing a relationship breakup). In total, our moderators explained less than 2% of variance in personality traits. Nonetheless, our study has several important implications for future research on individual differences in personality change. For example, future research should consider personal, environmental, and event-related moderators, use different analytical methods, and rely on highly powered samples to detect very small effects.
Collapse
Affiliation(s)
- Peter Haehner
- Department of Psychology, Ruhr University Bochum, Bochum, Germany
| | - Wiebke Bleidorn
- Department of Psychology, University of Zurich, Zurich, Switzerland
| | | |
Collapse
|
18
|
Breznau N, Rinke EM, Wuttke A, Nguyen HHV, Adem M, Adriaans J, Alvarez-Benjumea A, Andersen HK, Auer D, Azevedo F, Bahnsen O, Balzer D, Bauer G, Bauer PC, Baumann M, Baute S, Benoit V, Bernauer J, Berning C, Berthold A, Bethke FS, Biegert T, Blinzler K, Blumenberg JN, Bobzien L, Bohman A, Bol T, Bostic A, Brzozowska Z, Burgdorf K, Burger K, Busch KB, Carlos-Castillo J, Chan N, Christmann P, Connelly R, Czymara CS, Damian E, Ecker A, Edelmann A, Eger MA, Ellerbrock S, Forke A, Forster A, Gaasendam C, Gavras K, Gayle V, Gessler T, Gnambs T, Godefroidt A, Grömping M, Groß M, Gruber S, Gummer T, Hadjar A, Heisig JP, Hellmeier S, Heyne S, Hirsch M, Hjerm M, Hochman O, Hövermann A, Hunger S, Hunkler C, Huth N, Ignácz ZS, Jacobs L, Jacobsen J, Jaeger B, Jungkunz S, Jungmann N, Kauff M, Kleinert M, Klinger J, Kolb JP, Kołczyńska M, Kuk J, Kunißen K, Kurti Sinatra D, Langenkamp A, Lersch PM, Löbel LM, Lutscher P, Mader M, Madia JE, Malancu N, Maldonado L, Marahrens H, Martin N, Martinez P, Mayerl J, Mayorga OJ, McManus P, McWagner K, Meeusen C, Meierrieks D, Mellon J, Merhout F, Merk S, Meyer D, Micheli L, Mijs J, Moya C, Neunhoeffer M, Nüst D, Nygård O, Ochsenfeld F, Otte G, Pechenkina AO, Prosser C, Raes L, Ralston K, Ramos MR, Roets A, Rogers J, Ropers G, Samuel R, Sand G, Schachter A, Schaeffer M, Schieferdecker D, Schlueter E, Schmidt R, Schmidt KM, Schmidt-Catran A, Schmiedeberg C, Schneider J, Schoonvelde M, Schulte-Cloos J, Schumann S, Schunck R, Schupp J, Seuring J, Silber H, Sleegers W, Sonntag N, Staudt A, Steiber N, Steiner N, Sternberg S, Stiers D, Stojmenovska D, Storz N, Striessnig E, Stroppe AK, Teltemann J, Tibajev A, Tung B, Vagni G, Van Assche J, van der Linden M, van der Noll J, Van Hootegem A, Vogtenhuber S, Voicu B, Wagemans F, Wehl N, Werner H, Wiernik BM, Winter F, Wolf C, Yamada Y, Zhang N, Ziller C, Zins S, Żółtak T. Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. Proc Natl Acad Sci U S A 2022; 119:e2203150119. [PMID: 36306328 PMCID: PMC9636921 DOI: 10.1073/pnas.2203150119] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2022] [Accepted: 08/22/2022] [Indexed: 11/25/2022] Open
Abstract
This study explores how researchers' analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers' results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.
Collapse
Affiliation(s)
- Nate Breznau
- Research Center on Inequality and Social Policy (SOCIUM), University of Bremen, Bremen, 28359, Germany
| | - Eike Mark Rinke
- School of Politics and International Studies, University of Leeds, Leeds, LS2 9JT, United Kingdom
| | - Alexander Wuttke
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
- Department of Political Science, Ludwig Maximilian University, 80539 Munich, Germany
| | - Hung H. V. Nguyen
- Research Center on Inequality and Social Policy (SOCIUM), University of Bremen, Bremen, 28359, Germany
- Bremen International Graduate School of Social Sciences, 28359 Bremen, Germany
| | - Muna Adem
- Department of Sociology, Indiana University, Bloomington, IN 47405
| | - Jule Adriaans
- Socio-Economic Panel Study (SOEP), German Institute for Economic Research (DIW), 10117 Berlin, Germany
| | - Amalia Alvarez-Benjumea
- Mechanisms of Normative Change, Max Planck Institute for Research on Collective Goods, 53113 Bonn, Germany
| | - Henrik K. Andersen
- Institute of Sociology, Chemnitz University of Technology, 09126 Chemnitz, Germany
| | - Daniel Auer
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
| | - Flavio Azevedo
- Department of Psychology, University of Cambridge, Cambridge, CB23RQ, United Kingdom
| | - Oke Bahnsen
- School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany
| | - Dave Balzer
- Institute of Sociology, Johannes Gutenberg University Mainz, 55128 Mainz, Germany
| | - Gerrit Bauer
- Department of Sociology, Ludwig Maximilian University, 80801 Munich, Germany
| | - Paul C. Bauer
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
| | - Markus Baumann
- Heidelberg University, 69117 Heidelberg, Germany
- Institute for Political Science, Goethe University Frankfurt, 60323 Frankfurt, Germany
| | - Sharon Baute
- Comparative Political Economy, University of Konstanz, 78457 Konstanz, Germany
| | - Verena Benoit
- Department of Political Science, Ludwig Maximilian University, 80539 Munich, Germany
- Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, 96052 Bamberg, Germany
| | - Julian Bernauer
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
| | - Carl Berning
- Institute for Political Science, Johannes Gutenberg University Mainz, 55099 Mainz, Germany
| | - Anna Berthold
- Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, 96052 Bamberg, Germany
| | - Felix S. Bethke
- Research Department on Intrastate Conflict, Peace Research Institute Frankfurt, 60329 Frankfurt, Germany
| | - Thomas Biegert
- Department of Social Policy, London School of Economics and Political Science, London, WC2A 2AE, United Kingdom
| | - Katharina Blinzler
- Survey Data Curation, Leibniz Institute for the Social Sciences (GESIS), 50667 Cologne, Germany
| | - Johannes N. Blumenberg
- Knowledge Exchange and Outreach, Leibniz Institute for the Social Sciences (GESIS), 68159 Mannheim, Germany
| | - Licia Bobzien
- Jacques Delors Centre, Hertie School, 10117 Berlin, Germany
| | - Andrea Bohman
- Department of Sociology, Umeå University, 90187 Umeå, Sweden
| | - Thijs Bol
- Social Research Institute, Institute of Education, University College London, London, WC1H 0AL, United Kingdom
- Department of Sociology, University of Amsterdam, 1001 Amsterdam, The Netherlands
| | - Amie Bostic
- Department of Sociology, The University of Texas Rio Grande Valley, Brownsville, TX 78520
| | - Zuzanna Brzozowska
- Vienna Institute of Demography, Austrian Academy of Sciences, 1030 Vienna, Austria
- Austrian National Public Health Institute, Gesundheit Österreich (GÖG), 1030 Vienna, Austria
| | - Katharina Burgdorf
- School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany
| | - Kaspar Burger
- Social Research Institute, Institute of Education, University College London, London, WC1H 0AL, United Kingdom
- Department of Sociology, University of Zurich, 8050 Zurich, Switzerland
- Jacobs Center for Productive Youth, University of Zurich, 8050 Zurich, Switzerland
| | | | - Juan Carlos-Castillo
- Department of Sociology, University of Chile, Santiago, 7800284, Chile
- Center for Social Conflict and Cohesion Studies (COES), Pontificia Universidad Católica de Chile, Santiago, 8331150, Chile
| | - Nathan Chan
- Department of Political Science and International Relations, Loyola Marymount University, Los Angeles, CA 90045
| | - Pablo Christmann
- Data and Research on Society, Leibniz Institute for the Social Sciences, 68159 Mannheim, Germany
| | - Roxanne Connelly
- School of Social and Political Science, University of Edinburgh, Edinburgh, EH8 9LD, United Kingdom
| | | | - Elena Damian
- Lifestyle and Chronic Diseases, Epidemiology and Public Health, Sciensano, 1000 Brussels, Belgium
| | - Alejandro Ecker
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
| | | | - Maureen A. Eger
- Department of Sociology, Umeå University, 90187 Umeå, Sweden
| | - Simon Ellerbrock
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
- School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany
| | | | - Andrea Forster
- Empirical Educational and Higher Education Research, Freie Universität Berlin, 14195 Berlin, Germany
| | - Chris Gaasendam
- Department of Sociology, Center for Sociological Research, KU Leuven, 3000 Leuven, Belgium
| | - Konstantin Gavras
- School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany
| | - Vernon Gayle
- School of Social and Political Science, University of Edinburgh, Edinburgh, EH8 9LD, United Kingdom
| | - Theresa Gessler
- Kulturwissenschaftliche Fakultät, European University Viadrina, 15230 Frankfurt (Oder), Germany
| | - Timo Gnambs
- Educational Measurement, Leibniz Institute for Educational Trajectories, 96047 Bamberg, Germany
| | - Amélie Godefroidt
- Centre for Research on Peace and Development, KU Leuven, 3000 Leuven, Belgium
| | - Max Grömping
- School of Government and International Relations, Griffith University, Nathan, QLD, 4111, Australia
| | - Martin Groß
- Department of Sociology, University of Tübingen, 72074 Tübingen, Germany
| | - Stefan Gruber
- Max Planck Institute for Social Law and Social Policy, 80799 Munich, Germany
| | - Tobias Gummer
- Data and Research on Society, Leibniz Institute for the Social Sciences, 68159 Mannheim, Germany
| | - Andreas Hadjar
- University of Luxembourg, 4365 Esch-sur-Alzette, Luxembourg
- Wirtschafts- und Sozialwissenschaftliches Institut (WSI), Hans Böckler Foundation, 40474 Düsseldorf, Germany
- University of Fribourg, 1700 Fribourg, Switzerland
- Department of Social Sciences, University of Luxembourg, 4366 Esch-sur-Alzette, Luxembourg
| | - Jan Paul Heisig
- University of Groningen, 9712 CP Groningen,The Netherlands
- Research Group "Health and Social Inequality", Berlin Social Science Center (WZB), 10785 Berlin, Germany
| | - Sebastian Hellmeier
- Transformations of Democracy Unit, Berlin Social Science Center (WZB), 10785 Berlin, Germany
| | - Stefanie Heyne
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
| | - Magdalena Hirsch
- Research Unit Migration, Integration, Transnationalization, Berlin Social Science Center (WZB), 10785 Berlin, Germany
| | - Mikael Hjerm
- Department of Sociology, Umeå University, 90187 Umeå, Sweden
| | - Oshrat Hochman
- Data and Research on Society, Leibniz Institute for the Social Sciences, 68159 Mannheim, Germany
| | - Andreas Hövermann
- Wirtschafts- und Sozialwissenschaftliches Institut (WSI), Hans Böckler Foundation, 40474 Düsseldorf, Germany
- German Socio-Economic Panel Survey, 10117 Berlin, Germany
| | - Sophia Hunger
- Center for Civil Society Research, Berlin Social Science Center, 10785 Berlin, Germany
| | - Christian Hunkler
- Berlin Institute for Integration and Migration Research (BIM), Humboldt University Berlin, 10099 Berlin, Germany
| | - Nora Huth
- School of Human and Social Sciences, University of Wuppertal, 42119 Wuppertal, Germany
| | - Zsófia S. Ignácz
- Institute of Sociology, Goethe University Frankfurt, 60323 Frankfurt, Germany
| | - Laura Jacobs
- Department of Political Science, Université Libre de Bruxelles, 1050 Bruxelles, Belgium
| | - Jannes Jacobsen
- Zeppelin University, 88045 Friedrichshafen, Germany
- Cluster "Data-Methods-Monitoring", German Center for Integration and Migration Research (DeZIM),10117 Berlin, Germany
| | - Bastian Jaeger
- Department of Social Psychology, Tilburg University, 5037AB Tilburg, The Netherlands
| | - Sebastian Jungkunz
- Institute for Socio-Economics, University of Duisburg-Essen, 47057 Duisburg, Germany
- Institute of Political Science, University of Münster, 48149 Münster, Germany
- Chair of Political Sociology, University of Bamberg, 96052 Bamberg, Germany
| | - Nils Jungmann
- Survey Data Curation, Leibniz Institute for the Social Sciences (GESIS), 50667 Cologne, Germany
| | - Mathias Kauff
- Department of Psychology, Medical School Hamburg, 20457 Hamburg, Germany
| | - Manuel Kleinert
- Institute of Sociology, Justus Liebig University of Giessen, 35394 Giessen, Germany
| | - Julia Klinger
- Institute of Sociology and Social Psychology, University of Cologne, 50931 Cologne, Germany
| | - Jan-Philipp Kolb
- Federal Statistics Office Germany, Destatis, 65189 Wiesbaden, Germany
| | - Marta Kołczyńska
- Department of Research on Social and Institutional Transformations, Institute of Political Studies of the Polish Academy of Sciences, 00-625 Warsaw, Poland
| | - John Kuk
- Department of Political Science, University of Oklahoma, Norman, OK 73019
| | - Katharina Kunißen
- Institute of Sociology, Johannes Gutenberg University Mainz, 55128 Mainz, Germany
| | | | | | - Philipp M. Lersch
- Socio-Economic Panel Study (SOEP), German Institute for Economic Research (DIW), 10117 Berlin, Germany
- Department of Social Sciences, Humboldt University Berlin, 10099 Berlin, Germany
| | - Lea-Maria Löbel
- Socio-Economic Panel Study (SOEP), German Institute for Economic Research (DIW), 10117 Berlin, Germany
| | - Philipp Lutscher
- Department of Political Science, University of Oslo, 0851 Oslo, Norway
| | - Matthias Mader
- Department of Politics and Public Administration, University of Konstanz, 78457 Konstanz, Germany
| | - Joan E. Madia
- Department of Sociology, Nuffield College, University of Oxford, Oxford, OX1 1JD, United Kingdom
- Institute for the Evaluation of Public Policies, Fondazione Bruno Kessler, 38122 Trento, Italy
| | - Natalia Malancu
- The Institute of Citizenship Studies (InCite), University of Geneva, 1205 Geneva, Switzerland
| | - Luis Maldonado
- Instituto de Sociologia, Pontifical Catholic University of Chile, Santiago, 7820436, Chile
| | - Helge Marahrens
- Department of Sociology, Indiana University, Bloomington, IN 47405
| | - Nicole Martin
- Department of Politics, University of Manchester, Manchester, M19 2JS, United Kingdom
| | - Paul Martinez
- Department of Institutional Research, Western Governors University, Salt Lake City, UT 84107
| | - Jochen Mayerl
- Institute of Sociology, Chemnitz University of Technology, 09126 Chemnitz, Germany
| | - Oscar J. Mayorga
- Department of Sociology, University of California, Los Angeles, CA 90095
| | - Patricia McManus
- Department of Sociology, Indiana University, Bloomington, IN 47405
| | - Kyle McWagner
- Department of Political Science, The University of California, Irvine, CA 92617
| | - Cecil Meeusen
- Department of Sociology, Center for Sociological Research, KU Leuven, 3000 Leuven, Belgium
| | - Daniel Meierrieks
- Research Unit Migration, Integration, Transnationalization, Berlin Social Science Center (WZB), 10785 Berlin, Germany
| | - Jonathan Mellon
- Department of Politics, University of Manchester, Manchester, M19 2JS, United Kingdom
| | - Friedolin Merhout
- Department of Sociology and Centre for Social Data Science, University of Copenhagen, 1353 Copenhagen, Denmark
| | - Samuel Merk
- Department of School Development, University of Education Karlsruhe, 76133 Karlsruhe, Germany
| | - Daniel Meyer
- Department of Education and Social Sciences, University of Cologne, 50931 Cologne, Germany
| | - Leticia Micheli
- Department of Psychology III, Julius-Maximilians University Würzburg, 97070 Würzburg, Germany
| | - Jonathan Mijs
- Department of Sociology, Boston University, Boston, MA 02215
| | - Cristóbal Moya
- Faculty of Sociology, Bielefeld University, 33615 Bielefeld, Germany
| | - Marcel Neunhoeffer
- School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany
| | - Daniel Nüst
- Department of Geosciences, University of Münster, 49149 Münster, Germany
| | - Olav Nygård
- Division of Migration, Ethnicity and Society (REMESO), Linköping University, 60174 Linköping, Sweden
| | - Fabian Ochsenfeld
- Administrative Headquarters, Max Planck Society, 80539 Berlin, Germany
| | - Gunnar Otte
- Institute of Sociology, Johannes Gutenberg University Mainz, 55128 Mainz, Germany
| | | | - Christopher Prosser
- Department of Politics, International Relations and Philosophy, Royal Holloway University of London, London, TW20 0EX, United Kingdom
| | - Louis Raes
- Department of Economics, Tilburg University, 5037AB Tilburg, The Netherlands
| | - Kevin Ralston
- School of Social and Political Science, University of Edinburgh, Edinburgh, EH8 9LD, United Kingdom
| | - Miguel R. Ramos
- Department of Social Policy, Sociology and Criminology, University of Birmingham, Birmingham, B15 2TT, United Kingdom
| | - Arne Roets
- Department of Developmental, Personality and Social Psychology, Ghent University, 9000 Ghent, Belgium
| | - Jonathan Rogers
- Division of Social Science, New York University Abu Dhabi, Abu Dhabi, 10276, United Arab Emirates
| | - Guido Ropers
- School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany
| | - Robin Samuel
- University of Luxembourg, 4365 Esch-sur-Alzette, Luxembourg
- Department of Social Sciences, University of Luxembourg, 4366 Esch-sur-Alzette, Luxembourg
| | - Gregor Sand
- Max Planck Institute for Social Law and Social Policy, 80799 Munich, Germany
| | - Ariela Schachter
- Department of Sociology, Washington University in St. Louis, St. Louis, MO 63130
| | - Merlin Schaeffer
- Department of Sociology, University of Copenhagen, 1353 Copenhagen, Denmark
| | - David Schieferdecker
- Institute for Media and Communication Studies, Freie Universität Berlin, 14195 Berlin, Germany
| | - Elmar Schlueter
- Institute of Sociology, Justus Liebig University of Giessen, 35394 Giessen, Germany
| | - Regine Schmidt
- Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, 96052 Bamberg, Germany
| | - Katja M. Schmidt
- Socio-Economic Panel Study (SOEP), German Institute for Economic Research (DIW), 10117 Berlin, Germany
| | | | | | - Jürgen Schneider
- Tübingen School of Education, University of Tübingen, 72074 Tübingen, Germany
| | - Martijn Schoonvelde
- University College Dublin, Dublin 4, Ireland
- Department of European Languages and Cultures, University of Groningen, 9712 EK Groningen, The Netherlands
| | - Julia Schulte-Cloos
- Robert Schuman Center for Advanced Studies, European University Institute, 50133 Florence, Italy
| | - Sandy Schumann
- Department of Security and Crime Science, University College London, London,WC1E 6BT, United Kingdom
| | - Reinhard Schunck
- School of Human and Social Sciences, University of Wuppertal, 42119 Wuppertal, Germany
| | - Jürgen Schupp
- Socio-Economic Panel Study (SOEP), German Institute for Economic Research (DIW), 10117 Berlin, Germany
| | - Julian Seuring
- Department of Migration, Leibniz Institute for Educational Trajectories, 96047 Bamberg, Germany
| | - Henning Silber
- Department of Survey Design and Methodology, Leibniz Institute for the Social Sciences (GESIS), 68159 Mannheim, Germany
| | - Willem Sleegers
- Department of Social Psychology, Tilburg University, 5037AB Tilburg, The Netherlands
| | - Nico Sonntag
- Institute of Sociology, Johannes Gutenberg University Mainz, 55128 Mainz, Germany
| | | | - Nadia Steiber
- Department of Sociology, University of Vienna, 1090 Vienna, Austria
| | - Nils Steiner
- Institute for Political Science, Johannes Gutenberg University Mainz, 55099 Mainz, Germany
| | | | - Dieter Stiers
- Center for Political Science Research, KU Leuven, 3000 Leuven, Belgium
| | - Dragana Stojmenovska
- Department of Sociology, University of Amsterdam, 1001 Amsterdam, The Netherlands
| | - Nora Storz
- Interdisciplinary Social Science, Utrecht University, 3584 Utrecht, The Netherlands
| | - Erich Striessnig
- Department of Demography, University of Vienna, 1010 Vienna, Austria
| | - Anne-Kathrin Stroppe
- Survey Data Curation, Leibniz Institute for the Social Sciences (GESIS), 50667 Cologne, Germany
| | - Janna Teltemann
- Institute for Social Sciences, University of Hildesheim, 31141 Hildesheim, Germany
| | - Andrey Tibajev
- Division of Migration, Ethnicity and Society (REMESO), Linköping University, 60174 Linköping, Sweden
| | - Brian Tung
- Department of Sociology, Washington University in St. Louis, St. Louis, MO 63130
| | - Giacomo Vagni
- Social Research Institute, Institute of Education, University College London, London, WC1H 0AL, United Kingdom
| | - Jasper Van Assche
- Department of Developmental, Personality and Social Psychology, Ghent University, 9000 Ghent, Belgium
- Center for Social and Cultural Psychology, Université Libre de Bruxelles, 1050 Brussels, Belgium
| | - Meta van der Linden
- Interdisciplinary Social Science, Utrecht University, 3584 Utrecht, The Netherlands
| | | | - Arno Van Hootegem
- Department of Sociology, Center for Sociological Research, KU Leuven, 3000 Leuven, Belgium
| | - Stefan Vogtenhuber
- Education and Employment, Institute for Advanced Studies, University of Vienna, Vienna, 1080 Austria
| | - Bogdan Voicu
- Research Institute for Quality of Life, Romanian Academy, 010071 Bucharest, Romania
- Department of Sociology, Lucian Blaga University of Sibiu, 550024 Sibiu, Romania
| | - Fieke Wagemans
- Netherlands Institute for Social Research, 2500 BD The Hague, the Netherlands
- Policy Perspectives, Citizen Perspectives, and Behaviors, Netherlands Institute for Social Research, 2594 The Hague, The Netherlands
| | - Nadja Wehl
- Research Cluster "The Politics of Inequality", University of Konstanz, 78464 Konstanz, Germany
| | - Hannah Werner
- Center for Political Science Research, KU Leuven, 3000 Leuven, Belgium
| | | | - Fabian Winter
- Mechanisms of Normative Change, Max Planck Institute for Research on Collective Goods, 53113 Bonn, Germany
| | - Christof Wolf
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
- School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany
- President, Leibniz Institute for the Social Sciences (GESIS), 68159 Mannheim, Germany
| | - Yuki Yamada
- Faculty of Arts and Science, Kyushu University, Fukuoka, 819-0395, Japan
| | - Nan Zhang
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
| | - Conrad Ziller
- Institute for Socio-Economics, University of Duisburg-Essen, 47057 Duisburg, Germany
- Department of Political Science, University of Duisburg-Essen, 47057 Duisburg, Germany
| | - Stefan Zins
- Institute for Employment Research, Federal Employment Agency, 90478 Nuremberg, Germany
| | - Tomasz Żółtak
- Department of Research on Social and Institutional Transformations, Institute of Political Studies of the Polish Academy of Sciences, 00-625 Warsaw, Poland
| |
Collapse
|
19
|
Edelsbrunner PA, Ruggeri K, Damnjanović K, Greiff S, Lemoine JE, Ziegler M. Generalizability, Replicability, and New Insights Derived From Registered Reports Within Understudied Populations. EUROPEAN JOURNAL OF PSYCHOLOGICAL ASSESSMENT 2022. [DOI: 10.1027/1015-5759/a000743] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Affiliation(s)
| | - Kai Ruggeri
- Mailman School of Public Health, Columbia University, New York City, NY, USA
| | - Kaja Damnjanović
- Faculty of Philosophy, Laboratory for Experimental Psychology, Department of Psychology, Institute of Philosophy, University of Belgrade, Serbia
| | - Samuel Greiff
- Department of Behavioral and Cognitive Sciences, University of Luxembourg, Luxembourg
| | - Jérémy E. Lemoine
- Department of Psychology, University of East London, UK
- ESCP Business School, UK
| | | |
Collapse
|
20
|
Abstract
The extent to which results from complex datasets generalize across contexts is critically important to numerous scientific fields as well as to practitioners who rely on such analyses to guide important strategic decisions. Our initiative systematically investigated whether findings from the field of strategic management would emerge in new time periods and new geographies. Original findings that were statistically reliable in the first place were typically obtained again in novel tests, suggesting surprisingly little sensitivity to context. For some social scientific areas of inquiry, results from a specific time and place can be a meaningful guide as to what will be observed more generally. This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.
Collapse
|
21
|
Cyrus-Lai W, Tierney W, du Plessis C, Nguyen M, Schaerer M, Giulia Clemente E, Uhlmann EL. Avoiding Bias in the Search for Implicit Bias. PSYCHOLOGICAL INQUIRY 2022. [DOI: 10.1080/1047840x.2022.2106762] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/14/2022]
Affiliation(s)
| | | | | | - My Nguyen
- Lee Kong Chian School of Business, Singapore Management University, Singapore
| | - Michael Schaerer
- Lee Kong Chian School of Business, Singapore Management University, Singapore
| | | | | |
Collapse
|
22
|
Hanfstingl B. Future Objectivity Requires Perspective and Forward Combinatorial Meta-Analyses. Front Psychol 2022; 13:908311. [PMID: 35783689 PMCID: PMC9247499 DOI: 10.3389/fpsyg.2022.908311] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2022] [Accepted: 06/01/2022] [Indexed: 11/13/2022] Open
Abstract
This manuscript contributes to a future definition of objectivity by bringing together recent statements in epistemology and methodology. It outlines how improved objectivity can be achieved by systematically incorporating multiple perspectives, thereby improving the validity of science. The more result-biasing perspectives are known, the more a phenomenon of interest can be disentangled from these perspectives. Approaches that call for the integration of perspective into objectivity at the epistemological level or that systematically incorporate different perspectives at the statistical level already exist and are brought together in the manuscript. Recent developments in research methodology, such as transparency, reproducibility of research processes, pre-registration of studies, or free access to raw data, analysis strategies, and syntax, promote the explication of perspectives because they make the entire research process visible. How the explication of perspectives can be done practically is outlined in the manuscript. As a result, future research programs can be organized in such a way that meta-analyses and meta-meta-analyses can be conducted not only backward but forward and prospectively as a regular and thus well-prepared part of objectification and validation processes.
Collapse
Affiliation(s)
- Barbara Hanfstingl
- Institute for School and Instructional Development, University of Klagenfurt, Klagenfurt, Austria
| |
Collapse
|
23
|
Jacobs M, Remus A, Gaillard C, Menendez HM, Tedeschi LO, Neethirajan S, Ellis JL. ASAS-NANP symposium: mathematical modeling in animal nutrition: limitations and potential next steps for modeling and modelers in the animal sciences. J Anim Sci 2022; 100:skac132. [PMID: 35419602 PMCID: PMC9171330 DOI: 10.1093/jas/skac132] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2022] [Accepted: 04/08/2022] [Indexed: 11/12/2022] Open
Abstract
The field of animal science, and especially animal nutrition, relies heavily on modeling to accomplish its day-to-day objectives. New data streams ("big data") and the exponential increase in computing power have allowed the appearance of "new" modeling methodologies, under the umbrella of artificial intelligence (AI). However, many of these modeling methodologies have been around for decades. According to Gartner, technological innovation follows five distinct phases: technology trigger, peak of inflated expectations, trough of disillusionment, slope of enlightenment, and plateau of productivity. The appearance of AI certainly elicited much hype within agriculture leading to overpromised plug-and-play solutions in a field heavily dependent on custom solutions. The threat of failure can become real when advertising a disruptive innovation as sustainable. This does not mean that we need to abandon AI models. What is most necessary is to demystify the field and place a lesser emphasis on the technology and more on business application. As AI becomes increasingly more powerful and applications start to diverge, new research fields are introduced, and opportunities arise to combine "old" and "new" modeling technologies into hybrids. However, sustainable application is still many years away, and companies and universities alike do well to remain at the forefront. This requires investment in hardware, software, and analytical talent. It also requires a strong connection to the outside world to test, that which does, and does not work in practice and a close view of when the field of agriculture is ready to take its next big steps. Other research fields, such as engineering and automotive, have shown that the application power of AI can be far reaching but only if a realistic view of models as whole is maintained. In this review, we share our view on the current and future limitations of modeling and potential next steps for modelers in the animal sciences. First, we discuss the inherent dependencies and limitations of modeling as a human process. Then, we highlight how models, fueled by AI, can play an enhanced sustainable role in the animal sciences ecosystem. Lastly, we provide recommendations for future animal scientists on how to support themselves, the farmers, and their field, considering the opportunities and challenges the technological innovation brings.
Collapse
Affiliation(s)
- Marc Jacobs
- FR Analytics B.V., 7642 AP Wierden, The Netherlands
| | - Aline Remus
- Sherbrooke Research and Development Centre, Sherbrooke, QC J1M 1Z3, Canada
| | | | - Hector M Menendez
- Department of Animal Science, South Dakota State University, Rapid City, SD 57702, USA
| | - Luis O Tedeschi
- Department of Animal Science, Texas A&M University, College Station, TX 77843-2471, USA
| | - Suresh Neethirajan
- Farmworx, Adaptation Physiology, Animal Sciences Group, Wageningen University, 6700 AH, The Netherlands
| | - Jennifer L Ellis
- Department of Animal Biosciences, University of Guelph, Guelph, ON N1G 2W1, Canada
| |
Collapse
|
24
|
Exposing and overcoming the fixed-effect fallacy through crowd science. Behav Brain Sci 2022; 45:e8. [PMID: 35139965 DOI: 10.1017/s0140525x21000297] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
By organizing crowds of scientists to independently tackle the same research questions, we can collectively overcome the generalizability crisis. Strategies to draw inferences from a heterogeneous set of research approaches include aggregation, for instance, meta-analyzing the effect sizes obtained by different investigators, and parsing, attempting to identify theoretically meaningful moderators that explain the variability in results.
Collapse
|
25
|
Open Science at OBHDP. ORGANIZATIONAL BEHAVIOR AND HUMAN DECISION PROCESSES 2022. [DOI: 10.1016/j.obhdp.2021.104111] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
26
|
Paez A. Reproducibility of Research During COVID-19: Examining the Case of Population Density and the Basic Reproductive Rate from the Perspective of Spatial Analysis. GEOGRAPHICAL ANALYSIS 2021; 54:GEAN12307. [PMID: 34898693 PMCID: PMC8652856 DOI: 10.1111/gean.12307] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 06/09/2021] [Revised: 09/15/2021] [Accepted: 09/28/2021] [Indexed: 06/14/2023]
Abstract
The emergence of the novel SARS-CoV-2 coronavirus and the global COVID-19 pandemic in 2019 led to explosive growth in scientific research. Alas, much of the research in the literature lacks conditions to be reproducible, and recent publications on the association between population density and the basic reproductive number of SARS-CoV-2 are no exception. Relatively few papers share code and data sufficiently, which hinders not only verification but additional experimentation. In this article, an example of reproducible research shows the potential of spatial analysis for epidemiology research during COVID-19. Transparency and openness means that independent researchers can, with only modest efforts, verify findings and use different approaches as appropriate. Given the high stakes of the situation, it is essential that scientific findings, on which good policy depends, are as robust as possible; as the empirical example shows, reproducibility is one of the keys to ensure this.
Collapse
Affiliation(s)
- Antonio Paez
- School of EarthEnvironment and SocietyMcMaster UniversityHamiltonCanada
| |
Collapse
|
27
|
Aczel B, Szaszi B, Nilsonne G, van den Akker OR, Albers CJ, van Assen MALM, Bastiaansen JA, Benjamin D, Boehm U, Botvinik-Nezer R, Bringmann LF, Busch NA, Caruyer E, Cataldo AM, Cowan N, Delios A, van Dongen NNN, Donkin C, van Doorn JB, Dreber A, Dutilh G, Egan GF, Gernsbacher MA, Hoekstra R, Hoffmann S, Holzmeister F, Huber J, Johannesson M, Jonas KJ, Kindel AT, Kirchler M, Kunkels YK, Lindsay DS, Mangin JF, Matzke D, Munafò MR, Newell BR, Nosek BA, Poldrack RA, van Ravenzwaaij D, Rieskamp J, Salganik MJ, Sarafoglou A, Schonberg T, Schweinsberg M, Shanks D, Silberzahn R, Simons DJ, Spellman BA, St-Jean S, Starns JJ, Uhlmann EL, Wicherts J, Wagenmakers EJ. Consensus-based guidance for conducting and reporting multi-analyst studies. eLife 2021; 10:e72185. [PMID: 34751133 PMCID: PMC8626083 DOI: 10.7554/elife.72185] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2021] [Accepted: 11/07/2021] [Indexed: 11/13/2022] Open
Abstract
Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.
Collapse
Affiliation(s)
| | | | - Gustav Nilsonne
- Karolinska InstitutetStockholmSweden
- Stockholm UniversityStockholmSweden
| | | | | | | | - Jojanneke A Bastiaansen
- University Medical Center Groningen, University of GroningenGroningenNetherlands
- Friesland Mental Health Care ServicesLeeuwardenNetherlands
| | - Daniel Benjamin
- University of California Los AngelesLos AngelesUnited States
- National Bureau of Economic ResearchCambridgeUnited States
| | - Udo Boehm
- University of AmsterdamAmsterdamNetherlands
| | | | | | | | | | - Andrea M Cataldo
- McLean HospitalBelmontUnited States
- Harvard Medical SchoolBostonUnited States
| | | | | | | | | | | | - Anna Dreber
- Stockholm School of EconomicsStockholmSweden
- University of InnsbruckInnsbruckAustria
| | | | | | | | | | | | | | | | | | | | | | | | - Yoram K Kunkels
- University Medical Center Groningen, University of GroningenGroningenNetherlands
| | | | | | | | | | | | - Brian A Nosek
- Center for Open ScienceCharlottesvilleUnited States
- University of VirginiaCharlottesvilleUnited States
| | | | | | | | | | | | | | | | | | | | | | | | - Samuel St-Jean
- University of AlbertaEdmontonCanada
- Lund UniversityLundUnited States
| | | | | | | | | |
Collapse
|
28
|
Validation of Machine Learning Models for Structural Dam Behaviour Interpretation and Prediction. WATER 2021. [DOI: 10.3390/w13192717] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
The main aim of structural safety control is the multiple assessments of the expected dam behaviour based on models and the measurements and parameters that characterise the dam’s response and condition. In recent years, there is an increase in the use of data-based models for the analysis and interpretation of the structural behaviour of dams. Multiple Linear Regression is the conventional, widely used approach in dam engineering, although interesting results have been published based on machine learning algorithms such as artificial neural networks, support vector machines, random forest, and boosted regression trees. However, these models need to be carefully developed and properly assessed before their application in practice. This is even more relevant when an increase in users of machine learning models is expected. For this reason, this paper presents extensive work regarding the verification and validation of data-based models for the analysis and interpretation of observed dam’s behaviour. This is presented by means of the development of several machine learning models to interpret horizontal displacements in an arch dam in operation. Several validation techniques are applied, including historical data validation, sensitivity analysis, and predictive validation. The results are discussed and conclusions are drawn regarding the practical application of data-based models.
Collapse
|