1
|
Holzmeister F, Johannesson M, Böhm R, Dreber A, Huber J, Kirchler M. Heterogeneity in effect size estimates. Proc Natl Acad Sci U S A 2024; 121:e2403490121. [PMID: 39078672 PMCID: PMC11317577 DOI: 10.1073/pnas.2403490121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/19/2024] [Accepted: 06/28/2024] [Indexed: 07/31/2024] Open
Abstract
A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.
Collapse
Affiliation(s)
- Felix Holzmeister
- Department of Economics, University of Innsbruck, A-6020Innsbruck, Austria
| | - Magnus Johannesson
- Department of Economics, Stockholm School of Economics, SE-113 83Stockholm, Sweden
| | - Robert Böhm
- Department of Occupational, Economic, and Social Psychology, University of Vienna, A-1010Vienna, Austria
- Department of Psychology and Center for Social Data Science, University of Copenhagen, DK-1353Copenhagen, Denmark
| | - Anna Dreber
- Department of Economics, University of Innsbruck, A-6020Innsbruck, Austria
- Department of Economics, Stockholm School of Economics, SE-113 83Stockholm, Sweden
| | - Jürgen Huber
- Department of Banking and Finance, University of Innsbruck, A-6020Innsbruck, Austria
| | - Michael Kirchler
- Department of Banking and Finance, University of Innsbruck, A-6020Innsbruck, Austria
| |
Collapse
|
2
|
Purgar M, Glasziou P, Klanjscek T, Nakagawa S, Culina A. Supporting study registration to reduce research waste. Nat Ecol Evol 2024; 8:1391-1399. [PMID: 38839851 DOI: 10.1038/s41559-024-02433-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2023] [Accepted: 05/08/2024] [Indexed: 06/07/2024]
Abstract
An estimated 82-89% of ecological research and 85% of medical research has limited or no value to the end user because of various inefficiencies. We argue that registration and registered reports can enhance the quality and impact of ecological research. Drawing on evidence from other fields, chiefly medicine, we support our claim that registration can reduce research waste. However, increasing registration rates, quality and impact will be very slow without coordinated effort of funders, publishers and research institutions. We therefore call on them to facilitate the adoption of registration by providing adequate support. We outline several aspects to be considered when designing a registration system that would best serve the field of ecology. To further inform the development of such a system, we call for more research to identify the causes of low registration rates in ecology. We suggest short- and long-term actions to bolster registration and reduce research waste.
Collapse
Affiliation(s)
| | - Paul Glasziou
- Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Queensland, Australia
| | | | - Shinichi Nakagawa
- Evolution & Ecology Research Centre and School of Biological, Earth and Environmental Sciences, University of New South Wales, Sydney, New South Wales, Australia
- Theoretical Sciences Visiting Program, Okinawa Institute of Science and Technology Graduate University, Onna, Japan
| | - Antica Culina
- Ruđer Bošković Institute, Zagreb, Croatia.
- Netherlands Institute of Ecology, Royal Netherlands Academy of Arts and Sciences, Wageningen, the Netherlands.
| |
Collapse
|
3
|
Sarafoglou A, Hoogeveen S, van den Bergh D, Aczel B, Albers CJ, Althoff T, Botvinik-Nezer R, Busch NA, Cataldo AM, Devezer B, van Dongen NNN, Dreber A, Fried EI, Hoekstra R, Hoffman S, Holzmeister F, Huber J, Huntington-Klein N, Ioannidis J, Johannesson M, Kirchler M, Loken E, Mangin JF, Matzke D, Menkveld AJ, Nilsonne G, van Ravenzwaaij D, Schweinsberg M, Schulz-Kuempel H, Shanks DR, Simons DJ, Spellman BA, Stoevenbelt AH, Szaszi B, Trübutschek D, Tuerlinckx F, Uhlmann EL, Vanpaemel W, Wicherts J, Wagenmakers EJ. Subjective evidence evaluation survey for many-analysts studies. ROYAL SOCIETY OPEN SCIENCE 2024; 11:240125. [PMID: 39050728 PMCID: PMC11265885 DOI: 10.1098/rsos.240125] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/01/2024] [Accepted: 04/22/2024] [Indexed: 07/27/2024]
Abstract
Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.
Collapse
Affiliation(s)
| | | | - Don van den Bergh
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | - Balazs Aczel
- Institute of Psychology, ELTE Eötvös Lorénd University, Budapest, Hungary
| | - Casper J. Albers
- Heymans Institute for Psychological Research, University of Groningen, Groningen, The Netherlands
| | - Tim Althoff
- Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA
| | - Rotem Botvinik-Nezer
- Hebrew University of Jerusalem, Jerusalem, Israel
- Dartmouth College, Hanover, NH, USA
| | - Niko A. Busch
- Institute for Psychology, University of Münster, Münster, Germany
| | - Andrea M. Cataldo
- Center for Depression, Anxiety and Stress Research, McLean Hospital, Belmont, MA, USA
- Department of Psychiatry, Harvard Medical School, Boston, MA, USA
| | - Berna Devezer
- Department of Business, University of Idaho, Moscow, ID, USA
| | | | - Anna Dreber
- Stockholm School of Economics, Stockholm, Sweden
- University of Innsbruck, Innsbruck, Tirol, Austria
| | - Eiko I. Fried
- Department of Psychology, Leiden University, Leiden, The Netherlands
| | - Rink Hoekstra
- Nieuwenhuis Institute for Educational Research, University of Groningen, Groningen, The Netherlands
| | - Sabine Hoffman
- Department of Statistics, Ludwig-Maximilians-Universität München, Munchen, Bayern, Germany
| | | | - Jürgen Huber
- University of Innsbruck, Innsbruck, Tirol, Austria
| | | | - John Ioannidis
- Meta-Research Innovation Center at Stanford (METRICS) and Departments of Medicine, of Epidemiology and of Population Health, of Biomedical Data Science, and of Statistics, Stanford University, Stanford, CA, USA
| | | | | | - Eric Loken
- University of Conneticut, Storrs, CT, USA
| | - Jan-Francois Mangin
- University Paris-Saclay, Gif-sur-Yvette, France
- Neurospin CEA, Gif-sur-Yvette, Île-de-France, France
| | - Dora Matzke
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | | | | | - Don van Ravenzwaaij
- Heymans Institute for Psychological Research, University of Groningen, Groningen, The Netherlands
| | | | - Hannah Schulz-Kuempel
- Department of Statistics and The Institute for Medical Information Processing, Biometry, and Epidemiology, LMU Munich, Munchen, Bayern, Germany
- The Institute for Medical Information Processing, Biometry, and Epidemiology, LMU Munich, Munchen, Bayern, Germany
| | - David R. Shanks
- Division of Psychology and Language Sciences, University College London, 26 Bedford Way, London WC1H 0AP, UK
| | | | - Barbara A. Spellman
- School of Law, University of Virginia, 580 Massie Road, Charlottesville, VA, USA
| | - Andrea H. Stoevenbelt
- Nieuwenhuis Institute for Educational Research, University of Groningen, Groningen, The Netherlands
| | - Barnabas Szaszi
- Institute of Psychology, ELTE Eötvös Lorénd University, Budapest, Hungary
| | | | | | | | | | - Jelte Wicherts
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| | | |
Collapse
|
4
|
Martire KA, Chin JM, Davis C, Edmond G, Growns B, Gorski S, Kemp RI, Lee Z, Verdon CM, Jansen G, Lang T, Neal TM, Searston RA, Slocum J, Summersby S, Tangen JM, Thompson MB, Towler A, Watson D, Werrett MV, Younan M, Ballantyne KN. Understanding 'error' in the forensic sciences: A primer. Forensic Sci Int Synerg 2024; 8:100470. [PMID: 39005839 PMCID: PMC11240290 DOI: 10.1016/j.fsisyn.2024.100470] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2024] [Revised: 04/02/2024] [Accepted: 04/17/2024] [Indexed: 07/16/2024]
Abstract
This paper distils seven key lessons about 'error' from a collaborative webinar series between practitioners at Victoria Police Forensic Services Department and academics. It aims to provide the common understanding of error necessary to foster interdisciplinary dialogue, collaboration and research. The lessons underscore the inevitability, complexity and subjectivity of error, as well as opportunities for learning and growth. Ultimately, we argue that error can be a potent tool for continuous improvement and accountability, enhancing the reliability of forensic sciences and public trust. It is hoped the shared understanding provided by this paper will support future initiatives and funding for collaborative developments in this vital domain.
Collapse
Affiliation(s)
- Kristy A. Martire
- School of Psychology, University of New South Wales Sydney, Australia
| | | | - Carolyn Davis
- Major Crime Scene Unit, Victoria Police Forensic Services Department, Australia
| | - Gary Edmond
- School of Law, Society & Criminology, University of New South Wales Sydney, Australia
| | - Bethany Growns
- School of Psychology, Speech and Hearing, University of Canterbury, New Zealand
| | - Stacey Gorski
- Biological Sciences Group, Victoria Police Forensic Services Department, Australia
| | - Richard I. Kemp
- School of Psychology, University of New South Wales Sydney, Australia
| | - Zara Lee
- Fingerprint Sciences Group, Victoria Police Forensic Services Department, Australia
| | | | - Gabrielle Jansen
- Morwell Forensic Hub, Victoria Police Forensic Services Department, Australia
| | - Tanya Lang
- Major Crime Scene Unit, Victoria Police Forensic Services Department, Australia
| | | | | | - Joshua Slocum
- Fingerprint Sciences Group, Victoria Police Forensic Services Department, Australia
| | - Stephanie Summersby
- Office of the Chief Forensic Scientist, Victoria Police Forensic Services Department, Australia
| | - Jason M. Tangen
- School of Psychology, The University of Queensland, Australia
| | - Matthew B. Thompson
- School of Psychology, Murdoch University, Australia
- Centre for Biosecurity and One Health, Harry Butler Institute, Murdoch University, Australia
| | - Alice Towler
- School of Psychology, The University of Queensland, Australia
| | - Darren Watson
- Ballistics Unit, Victoria Police Forensic Services Department, Australia
| | - Melissa V. Werrett
- Chemical Trace Unit, Chemical and Physical Sciences Group, Victoria Police Forensic Services Department, Australia
| | - Mariam Younan
- School of Psychology, University of New South Wales Sydney, Australia
| | - Kaye N. Ballantyne
- Office of the Chief Forensic Scientist, Victoria Police Forensic Services Department, Australia
| |
Collapse
|
5
|
Schneider J, Striebing C, Hochfeld K, Lorenz T. Establishing circularity: development and validation of the circular work value scale (CWVS). Front Psychol 2024; 15:1296282. [PMID: 38646124 PMCID: PMC11026680 DOI: 10.3389/fpsyg.2024.1296282] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2024] [Accepted: 03/18/2024] [Indexed: 04/23/2024] Open
Abstract
Objectives Addressing the lack of German-language instruments, this study aims to develop a questionnaire that enables the measurement of work values. According to the theory of basic human values (Schwartz, 1992), a culturally fitting questionnaire is validated by covering constructs in the four broader dimensions of Social, Prestige, Intrinsic, and Extrinsic work values. Convergent, discriminant and incremental congruent validity are assessed. Method Data were collected in a cross-sectional online-based panel survey. Individuals working more than 20 h per week were included (N = 1,049). Using a genetic algorithm, an economical and valid questionnaire was designed to assess work values. Results The 11 work values are measurable with three items each. They provide a good fit to the data with support for strict measurement invariance. The empirical associations to estimate construct validity overall reflect expected relations to social and individualistic work motives, neuroticism, environmental awareness, and basic values. Furthermore, congruent incremental validity is supported with relations to value congruence of the person-organization fit, and multidimensional scaling supports the assumed theoretical circularity of the work values. Implications This study developed a questionnaire that enables a theory-based valid measurement of work values. The questionnaire allows practitioners to economically collect information about the value structure of employees or applicants. Future research should consider the development of work values over time and investigate whether more distinctive constructs provide a better fit in the nomological network.
Collapse
Affiliation(s)
- Jannick Schneider
- Center for Responsible Research and Innovation, Fraunhofer Institute for Industrial Engineering, Berlin, Germany
- Department of Psychology, MSB Medical School Berlin, Berlin, Germany
| | - Clemens Striebing
- Center for Responsible Research and Innovation, Fraunhofer Institute for Industrial Engineering, Berlin, Germany
| | - Katharina Hochfeld
- Center for Responsible Research and Innovation, Fraunhofer Institute for Industrial Engineering, Berlin, Germany
| | - Timo Lorenz
- Department of Psychology, MSB Medical School Berlin, Berlin, Germany
| |
Collapse
|
6
|
Thompson LDR. Ten Ways to Improve Getting a Scientific Manuscript Accepted. Head Neck Pathol 2024; 18:22. [PMID: 38503984 PMCID: PMC10951136 DOI: 10.1007/s12105-024-01617-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/01/2024] [Accepted: 01/14/2024] [Indexed: 03/21/2024]
Abstract
BACKGROUND Scientific publication is the cornerstone to academic and private practice advancement in patient management and outcomes. Writing a manuscript requires a certain discipline and skill set that can be achieved with diligence and hard work. METHODS Anecdotal and review. RESULTS Several factors must be considered in scientific writing and journal manuscript submission and acceptance. Choosing where to submit the manuscript; understanding the instructions to authors; disclosing ethically; formatting correctly; never plagiarizing; supplying high quality appropriate images; creating meaningful tables; curating a pertinent but thorough bibliography; having valid, supported conclusions; and respecting timelines. CONCLUSION A discussion of relevant components in manuscript writing and journal submission to improve your chances of acceptance.
Collapse
Affiliation(s)
- Lester D R Thompson
- Head and Neck Pathology Consultations, 22543 Ventura Blvd, Ste 220 PMB1034, Woodland Hills, CA, 91364, USA.
| |
Collapse
|
7
|
Neoh MJY, Carollo A, Lee A, Esposito G. Fifty years of research on questionable research practises in science: quantitative analysis of co-citation patterns. ROYAL SOCIETY OPEN SCIENCE 2023; 10:230677. [PMID: 37859842 PMCID: PMC10582594 DOI: 10.1098/rsos.230677] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/19/2023] [Accepted: 09/26/2023] [Indexed: 10/21/2023]
Abstract
Questionable research practises (QRPs) have been the focus of the scientific community amid greater scrutiny and evidence highlighting issues with replicability across many fields of science. To capture the most impactful publications and the main thematic domains in the literature on QRPs, this study uses a document co-citation analysis. The analysis was conducted on a sample of 341 documents that covered the past 50 years of research in QRPs. Nine major thematic clusters emerged. Statistical reporting and statistical power emerged as key areas of research, where systemic-level factors in how research is conducted are consistently raised as the precipitating factors for QRPs. There is also an encouraging shift in the focus of research into open science practises designed to address engagement in QRPs. Such a shift is indicative of the growing momentum of the open science movement, and more research can be conducted on how these practises are employed on the ground and how their uptake by researchers can be further promoted. However, the results suggest that, while pre-registration and registered reports receive the most research interest, less attention has been paid to other open science practises (e.g. data sharing).
Collapse
Affiliation(s)
- Michelle Jin Yee Neoh
- Psychology Program, School of Social Sciences, Nanyang Technological University, Singapore 639818, Singapore
| | - Alessandro Carollo
- Department of Psychology and Cognitive Science, University of Trento, Rovereto 38068, Italy
| | - Albert Lee
- Psychology Program, School of Social Sciences, Nanyang Technological University, Singapore 639818, Singapore
| | - Gianluca Esposito
- Department of Psychology and Cognitive Science, University of Trento, Rovereto 38068, Italy
| |
Collapse
|
8
|
Kostygina G, Kim Y, Seeskin Z, LeClere F, Emery S. Disclosure Standards for Social Media and Generative Artificial Intelligence Research: Toward Transparency and Replicability. SOCIAL MEDIA + SOCIETY 2023; 9:10.1177/20563051231216947. [PMID: 38239338 PMCID: PMC10795517 DOI: 10.1177/20563051231216947] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 01/22/2024]
Abstract
Social media dominate today's information ecosystem and provide valuable information for social research. Market researchers, social scientists, policymakers, government entities, public health researchers, and practitioners recognize the potential for social data to inspire innovation, support products and services, characterize public opinion, and guide decisions. The appeal of mining these rich datasets is clear. However, there is potential risk of data misuse, underscoring an equally huge and fundamental flaw in the research: there are no procedural standards and little transparency. Transparency across the processes of collecting and analyzing social media data is often limited due to proprietary algorithms. Spurious findings and biases introduced by artificial intelligence (AI) demonstrate the challenges this lack of transparency poses for research. Social media research remains a virtual "wild west," with no clear standards for reporting regarding data retrieval, preprocessing steps, analytic methods, or interpretation. Use of emerging generative AI technologies to augment social media analytics can undermine validity and replicability of findings, potentially turning this research into a "black box" enterprise. Clear guidance for social media analyses and reporting is needed to assure the quality of the resulting research. In this article, we propose criteria for evaluating the quality of studies using social media data, grounded in established scientific practice. We offer clear documentation guidelines to ensure that social data are used properly and transparently in research and applications. A checklist of disclosure elements to meet minimal reporting standards is proposed. These criteria will make it possible for scholars and practitioners to assess the quality, credibility, and comparability of research findings using digital data.
Collapse
|
9
|
Wyant K, Moshontz H, Ward SB, Fronk GE, Curtin JJ. Acceptability of Personal Sensing Among People With Alcohol Use Disorder: Observational Study. JMIR Mhealth Uhealth 2023; 11:e41833. [PMID: 37639300 PMCID: PMC10495858 DOI: 10.2196/41833] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/11/2022] [Revised: 03/14/2023] [Accepted: 07/25/2023] [Indexed: 08/29/2023] Open
Abstract
BACKGROUND Personal sensing may improve digital therapeutics for mental health care by facilitating early screening, symptom monitoring, risk prediction, and personalized adaptive interventions. However, further development and the use of personal sensing requires a better understanding of its acceptability to people targeted for these applications. OBJECTIVE We aimed to assess the acceptability of active and passive personal sensing methods in a sample of people with moderate to severe alcohol use disorder using both behavioral and self-report measures. This sample was recruited as part of a larger grant-funded project to develop a machine learning algorithm to predict lapses. METHODS Participants (N=154; n=77, 50% female; mean age 41, SD 11.9 years; n=134, 87% White and n=150, 97% non-Hispanic) in early recovery (1-8 weeks of abstinence) were recruited to participate in a 3-month longitudinal study. Participants were modestly compensated for engaging with active (eg, ecological momentary assessment [EMA], audio check-in, and sleep quality) and passive (eg, geolocation, cellular communication logs, and SMS text message content) sensing methods that were selected to tap into constructs from the Relapse Prevention model by Marlatt. We assessed 3 behavioral indicators of acceptability: participants' choices about their participation in the study at various stages in the procedure, their choice to opt in to provide data for each sensing method, and their adherence to a subset of the active methods (EMA and audio check-in). We also assessed 3 self-report measures of acceptability (interference, dislike, and willingness to use for 1 year) for each method. RESULTS Of the 192 eligible individuals screened, 191 consented to personal sensing. Most of these individuals (169/191, 88.5%) also returned 1 week later to formally enroll, and 154 participated through the first month follow-up visit. All participants in our analysis sample opted in to provide data for EMA, sleep quality, geolocation, and cellular communication logs. Out of 154 participants, 1 (0.6%) did not provide SMS text message content and 3 (1.9%) did not provide any audio check-ins. The average adherence rate for the 4 times daily EMA was .80. The adherence rate for the daily audio check-in was .54. Aggregate participant ratings indicated that all personal sensing methods were significantly more acceptable (all P<.001) compared with neutral across subjective measures of interference, dislike, and willingness to use for 1 year. Participants did not significantly differ in their dislike of active methods compared with passive methods (P=.23). However, participants reported a higher willingness to use passive (vs active) methods for 1 year (P=.04). CONCLUSIONS These results suggest that active and passive sensing methods are acceptable for people with alcohol use disorder over a longer period than has previously been assessed. Important individual differences were observed across people and methods, indicating opportunities for future improvement.
Collapse
Affiliation(s)
- Kendra Wyant
- Department of Psychology, University of Wisconsin-Madison, Madison, WI, United States
| | - Hannah Moshontz
- Department of Psychology, University of Wisconsin-Madison, Madison, WI, United States
| | - Stephanie B Ward
- Department of Psychology, University of Wisconsin-Madison, Madison, WI, United States
| | - Gaylen E Fronk
- Department of Psychology, University of Wisconsin-Madison, Madison, WI, United States
| | - John J Curtin
- Department of Psychology, University of Wisconsin-Madison, Madison, WI, United States
| |
Collapse
|
10
|
Compagnone D, Cecchinato R, Pezzi A, Langella F, Damilano M, Redaelli A, Vanni D, Lamartina C, Berjano P, Boriani S. Diagnostic Approach and Differences between Spinal Infections and Tumors. Diagnostics (Basel) 2023; 13:2737. [PMID: 37685273 PMCID: PMC10487270 DOI: 10.3390/diagnostics13172737] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2023] [Revised: 08/08/2023] [Accepted: 08/19/2023] [Indexed: 09/10/2023] Open
Abstract
STUDY DESIGN A systematic review of the literature about differential diagnosis between spine infection and bone tumors of the spine. BACKGROUND AND PURPOSE The differential diagnosis between spine infection and bone tumors of the spine can be misled by the prevalence of one of the conditions over the other in different areas of the world. A review of the existing literature on suggestive or even pathognomonic imaging aspects of both can be very useful for correctly orientating the diagnosis and deciding the most appropriate area for biopsy. The purpose of our study is to identify which imaging technique is the most reliable to suggest the diagnosis between spine infection and spine bone tumor. METHODS A primary search on Medline through PubMed distribution was made. We identified five main groups: tuberculous, atypical spinal tuberculosis, pyogenic spondylitis, and neoplastic (primitive and metastatic). For each group, we evaluated the commonest localization, characteristics at CT, CT perfusion, MRI, MRI with Gadolinium, MRI diffusion (DWI) and, in the end, the main features for each group. RESULTS A total of 602 studies were identified through the database search and a screening by titles and abstracts was performed. After applying inclusion and exclusion criteria, 34 articles were excluded and a total of 22 full-text articles were assessed for eligibility. For each article, the role of CT-scan, CT-perfusion, MRI, MRI with Gadolinium and MRI diffusion (DWI) in distinguishing the most reliable features to suggest the diagnosis of spine infection versus bone tumor/metastasis was collected. CONCLUSION Definitive differential diagnosis between infection and tumor requires biopsy and culture. The sensitivity and specificity of percutaneous biopsy are 72% and 94%, respectively. Imaging studies can be added to address the diagnosis, but a multidisciplinary discussion with radiologists and nuclear medicine specialists is mandatory.
Collapse
Affiliation(s)
| | | | - Andrea Pezzi
- IRCCS Ospedale Galeazzi—Sant’Ambrogio, 20157 Milan, Italy
- Residency Program in Orthopaedics and Traumatology, University of Milan, 20141 Milan, Italy
| | | | - Marco Damilano
- IRCCS Ospedale Galeazzi—Sant’Ambrogio, 20157 Milan, Italy
| | | | - Daniele Vanni
- IRCCS Ospedale Galeazzi—Sant’Ambrogio, 20157 Milan, Italy
| | | | - Pedro Berjano
- IRCCS Ospedale Galeazzi—Sant’Ambrogio, 20157 Milan, Italy
| | | |
Collapse
|
11
|
Jensen UT, Rohner D, Bornet O, Carron D, Garner P, Loupi D, Antonakis J. Combating COVID-19 with charisma: Evidence on governor speeches in the United States. THE LEADERSHIP QUARTERLY 2023:101702. [PMID: 37361053 PMCID: PMC10201331 DOI: 10.1016/j.leaqua.2023.101702] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/09/2021] [Revised: 04/22/2023] [Accepted: 05/02/2023] [Indexed: 06/28/2023]
Abstract
Using field and laboratory data, we show that leader charisma can affect COVID-related mitigating behaviors. We coded a panel of U.S. governor speeches for charisma signaling using a deep neural network algorithm. The model explains variation in stay-at-home behavior of citizens based on their smart phone data movements, showing a robust effect of charisma signaling: stay-at-home behavior increased irrespective of state-level citizen political ideology or governor party allegiance. Republican governors with a particularly high charisma signaling score impacted the outcome more relative to Democratic governors in comparable conditions. Our results also suggest that one standard deviation higher charisma signaling in governor speeches could potentially have saved 5,350 lives during the study period (02/28/2020-05/14/2020). Next, in an incentivized laboratory experiment we found that politically conservative individuals are particularly prone to believe that their co-citizens will follow governor appeals to distance or stay at home when exposed to a speech that is high in charisma; these beliefs in turn drive their preference to engage in those behaviors. These results suggest that political leaders should consider additional "soft-power" levers like charisma-which can be learned-to complement policy interventions for pandemics or other public heath crises, especially with certain populations who may need a "nudge."
Collapse
Affiliation(s)
- Ulrich Thy Jensen
- Center for Organization Research and Design, School of Public Affairs, Arizona State University, United States
- Crown Prince Frederik Center for Public Leadership, Aarhus University, Denmark
| | - Dominic Rohner
- Faculty of Business and Economics (HEC Lausanne), University of Lausanne & E4S, Centre for Economic Policy Research (CEPR), Switzerland
| | | | | | | | | | - John Antonakis
- Faculty of Business and Economics (HEC Lausanne), University of Lausanne & E4S, Switzerland
| |
Collapse
|
12
|
Mah EY, Grannon KEL, Campbell A, Tamburri N, Jamieson RK, Lindsay DS. A direct replication and extension of Popp and Serra (2016, experiment 1): better free recall and worse cued recall of animal names than object names, accounting for semantic similarity. Front Psychol 2023; 14:1146200. [PMID: 37275705 PMCID: PMC10232972 DOI: 10.3389/fpsyg.2023.1146200] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2023] [Accepted: 04/13/2023] [Indexed: 06/07/2023] Open
Abstract
Introduction Free recall tends to be better for names of animate concepts such as animals than for names of inanimate objects. In Popp and Serra's 2016 article, the authors replicated this "animacy effect" in free recall but when participants studied words in pairs (animate-animate pairs intermixed with inanimate-inanimate pairs) and were tested with cued recall, performance was better for inanimate-inanimate pairs than for animate-animate pairs ("reverse animacy"). We tested the replicability of this surprising effect and one possible explanation for the effect (semantic similarity). Methods Our Experiment 1 was a preregistered direct replication (N = 101) of Popp and Serra's Experiment 1 (mixed-lists condition). In a second preregistered experiment conducted in four different samples (undergraduate N = 153, undergraduate N = 143, online Prolific N = 101, online Prolific/English-as-a-first-language N = 150), we manipulated the within-category semantic similarity of animal and object wordlists. Results AIn Experiment 1, just as in Popp and Serra, we observed an animacy effect for free recall and a reverse animacy effect for cued recall. Unlike Popp and Serra, we found that controlling for interference effects rendered the reverse animacy effect non-significant. We took this as evidence that characteristics of the stimulus sets (e.g., category structure, within-category similarity) may play a role in animacy and reverse animacy effects. In Experiment 2, in three out of our four samples, we observed reverse animacy effects when within-category similarity was higher for animals and when within-category similarity was equated for animals and objects. Discussion Our results suggest that the reverse animacy effect observed in Popp and Serra's 2016 article is a robust and replicable effect, but that semantic similarity alone cannot explain the effect.
Collapse
Affiliation(s)
- Eric Y. Mah
- Department of Psychology, University of Victoria, Victoria, BC, Canada
| | | | - Alison Campbell
- Department of Psychology, University of Victoria, Victoria, BC, Canada
| | - Nicholas Tamburri
- Department of Psychology, University of Victoria, Victoria, BC, Canada
| | | | | |
Collapse
|
13
|
Schroeder PA, Artemenko C, Kosie JE, Cockx H, Stute K, Pereira J, Klein F, Mehler DMA. Using preregistration as a tool for transparent fNIRS study design. NEUROPHOTONICS 2023; 10:023515. [PMID: 36908680 PMCID: PMC9993433 DOI: 10.1117/1.nph.10.2.023515] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/09/2022] [Accepted: 01/11/2023] [Indexed: 05/04/2023]
Abstract
Significance The expansion of functional near-infrared spectroscopy (fNIRS) methodology and analysis tools gives rise to various design and analytical decisions that researchers have to make. Several recent efforts have developed guidelines for preprocessing, analyzing, and reporting practices. For the planning stage of fNIRS studies, similar guidance is desirable. Study preregistration helps researchers to transparently document study protocols before conducting the study, including materials, methods, and analyses, and thus, others to verify, understand, and reproduce a study. Preregistration can thus serve as a useful tool for transparent, careful, and comprehensive fNIRS study design. Aim We aim to create a guide on the design and analysis steps involved in fNIRS studies and to provide a preregistration template specified for fNIRS studies. Approach The presented preregistration guide has a strong focus on fNIRS specific requirements, and the associated template provides examples based on continuous-wave (CW) fNIRS studies conducted in humans. These can, however, be extended to other types of fNIRS studies. Results On a step-by-step basis, we walk the fNIRS user through key methodological and analysis-related aspects central to a comprehensive fNIRS study design. These include items specific to the design of CW, task-based fNIRS studies, but also sections that are of general importance, including an in-depth elaboration on sample size planning. Conclusions Our guide introduces these open science tools to the fNIRS community, providing researchers with an overview of key design aspects and specification recommendations for comprehensive study planning. As such it can be used as a template to preregister fNIRS studies or merely as a tool for transparent fNIRS study design.
Collapse
Affiliation(s)
- Philipp A. Schroeder
- University of Tuebingen, Department of Psychology, Faculty of Science, Tuebingen, Germany
| | - Christina Artemenko
- University of Tuebingen, Department of Psychology, Faculty of Science, Tuebingen, Germany
| | - Jessica E. Kosie
- Princeton University, Social and Natural Sciences, Department of Psychology, Princeton, New Jersey, United States
| | - Helena Cockx
- Radboud University, Donders Institute for Brain, Cognition and Behaviour, Biophysics Department, Faculty of Science, Nijmegen, The Netherlands
| | - Katharina Stute
- Chemnitz University of Technology, Institute of Human Movement Science and Health, Faculty of Behavioural and Social Sciences, Chemnitz, Germany
| | - João Pereira
- University of Coimbra, Coimbra Institute for Biomedical Imaging and Translational Research, Coimbra, Portugal
| | - Franziska Klein
- University of Oldenburg, Department of Psychology, Neurocognition and functional Neurorehabilitation Group, Oldenburg (Oldb), Germany
- RWTH Aachen University, Medical School, Department of Psychiatry, Psychotherapy and Psychosomatics, Aachen, Germany
| | - David M. A. Mehler
- RWTH Aachen University, Medical School, Department of Psychiatry, Psychotherapy and Psychosomatics, Aachen, Germany
- University of Münster, Institute for Translational Psychiatry, Medical School, Münster, Germany
| |
Collapse
|
14
|
Wulff JN, Sajons GB, Pogrebna G, Lonati S, Bastardoz N, Banks GC, Antonakis J. Common methodological mistakes. THE LEADERSHIP QUARTERLY 2023. [DOI: 10.1016/j.leaqua.2023.101677] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/23/2023]
|
15
|
Kekecs Z, Palfi B, Szaszi B, Szecsi P, Zrubka M, Kovacs M, Bakos BE, Cousineau D, Tressoldi P, Schmidt K, Grassi M, Evans TR, Yamada Y, Miller JK, Liu H, Yonemitsu F, Dubrov D, Röer JP, Becker M, Schnepper R, Ariga A, Arriaga P, Oliveira R, Põldver N, Kreegipuu K, Hall B, Wiechert S, Verschuere B, Girán K, Aczel B. Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project. ROYAL SOCIETY OPEN SCIENCE 2023; 10:191375. [PMID: 36756055 PMCID: PMC9890107 DOI: 10.1098/rsos.191375] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/07/2019] [Accepted: 01/11/2023] [Indexed: 06/18/2023]
Abstract
The low reproducibility rate in social sciences has produced hesitation among researchers in accepting published findings at their face value. Despite the advent of initiatives to increase transparency in research reporting, the field is still lacking tools to verify the credibility of research reports. In the present paper, we describe methodologies that let researchers craft highly credible research and allow their peers to verify this credibility. We demonstrate the application of these methods in a multi-laboratory replication of Bem's Experiment 1 (Bem 2011 J. Pers. Soc. Psychol. 100, 407-425. (doi:10.1037/a0021524)) on extrasensory perception (ESP), which was co-designed by a consensus panel including both proponents and opponents of Bem's original hypothesis. In the study we applied direct data deposition in combination with born-open data and real-time research reports to extend transparency to protocol delivery and data collection. We also used piloting, checklists, laboratory logs and video-documented trial sessions to ascertain as-intended protocol delivery, and external research auditors to monitor research integrity. We found 49.89% successful guesses, while Bem reported 53.07% success rate, with the chance level being 50%. Thus, Bem's findings were not replicated in our study. In the paper, we discuss the implementation, feasibility and perceived usefulness of the credibility-enhancing methodologies used throughout the project.
Collapse
Affiliation(s)
- Zoltan Kekecs
- Institute of Psychology, ELTE, Eotvos Lorand University, Izabella u 46. 1064, Budapest, Hungary
- Department of Psychology, Lund University, Box 213, Lund 221 00, Sweden
| | - Bence Palfi
- Department of Surgery and Cancer, Imperial College London, London, UK
| | - Barnabas Szaszi
- Institute of Psychology, ELTE, Eotvos Lorand University, Izabella u 46. 1064, Budapest, Hungary
| | - Peter Szecsi
- Institute of Psychology, ELTE, Eotvos Lorand University, Izabella u 46. 1064, Budapest, Hungary
| | - Mark Zrubka
- Department of Psychology, University of Amsterdam, P.O. Box 19268, 1000 GG Amsterdam, The Netherlands
| | - Marton Kovacs
- Institute of Psychology, ELTE, Eotvos Lorand University, Izabella u 46. 1064, Budapest, Hungary
| | - Bence E. Bakos
- Institute of Psychology, ELTE, Eotvos Lorand University, Izabella u 46. 1064, Budapest, Hungary
| | - Denis Cousineau
- École de psychologie, University of Ottawa, 136, Jean-Jacques Lussier, Ontario, Canada, K1N 6N5
| | - Patrizio Tressoldi
- Studium Patavinum, Università di Padova via Venezia 8, 35131 Padova, Italy
| | - Kathleen Schmidt
- Department of Psychology, Ashland University, Ashland, OH 44805, USA
- School of Psychological & Behavioral Sciences, Southern Illinois University, Carbondale, USA
| | - Massimo Grassi
- Dipartimento di Psicologia Generale, Università di Padova via Venezia 8, 35131 Padova, Italy
| | | | - Yuki Yamada
- Faculty of Arts and Science, Kyushu University, 744 Motooka, Nishi-ku, Fukuoka 819-0395, Japan
| | - Jeremy K. Miller
- Department of Psychology, Willamette University, 900 State Street, Salem, OR 97301, USA
| | - Huanxu Liu
- Graduate School of Human-Environment Studies, Kyushu University, 744 Motooka, Nishi-ku, Fukuoka 819-0395, Japan
| | - Fumiya Yonemitsu
- Faculty of Letters, Chuo University, Hachioji, Tokyo 192-0393, Japan
| | - Dmitrii Dubrov
- National Research University Higher School of Economics, Russian Federation
| | - Jan Philipp Röer
- Department of Psychology and Psychotherapy, Witten/Herdecke University, Witten, Germany
| | - Marvin Becker
- Department of Psychology and Psychotherapy, Witten/Herdecke University, Witten, Germany
| | - Roxane Schnepper
- Department of Psychology and Psychotherapy, Witten/Herdecke University, Witten, Germany
| | - Atsunori Ariga
- Faculty of Letters, Chuo University, Hachioji, Tokyo 192-0393, Japan
| | - Patrícia Arriaga
- Iscte-University Institute of Lisbon, CIS_Iscte, Av. das Forças Armadas, 1649-026, Lisbon, Portugal
| | - Raquel Oliveira
- Iscte-University Institute of Lisbon, CIS_Iscte, Av. das Forças Armadas, 1649-026, Lisbon, Portugal
| | - Nele Põldver
- University of Tartu Institute of Psychology, Estonia
| | | | - Braeden Hall
- School of Psychological & Behavioral Sciences, Southern Illinois University, Carbondale, USA
| | - Sera Wiechert
- University of Amsterdam, Amsterdam, The Netherlands
- Hebrew University of Jerusalem, Jerusalem, Israel
| | | | - Kyra Girán
- Institute of Psychology, ELTE, Eotvos Lorand University, Izabella u 46. 1064, Budapest, Hungary
| | - Balazs Aczel
- Institute of Psychology, ELTE, Eotvos Lorand University, Izabella u 46. 1064, Budapest, Hungary
| |
Collapse
|
16
|
Pittelkow MM, Field SM, Isager PM, van’t Veer AE, Anderson T, Cole SN, Dominik T, Giner-Sorolla R, Gok S, Heyman T, Jekel M, Luke TJ, Mitchell DB, Peels R, Pendrous R, Sarrazin S, Schauer JM, Specker E, Tran US, Vranka MA, Wicherts JM, Yoshimura N, Zwaan RA, van Ravenzwaaij D. The process of replication target selection in psychology: what to consider? ROYAL SOCIETY OPEN SCIENCE 2023; 10:210586. [PMID: 36756069 PMCID: PMC9890109 DOI: 10.1098/rsos.210586] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/07/2021] [Accepted: 01/13/2023] [Indexed: 06/18/2023]
Abstract
Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well-justified, systematic and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of individuals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target. The resulting checklist can be used for transparently communicating the rationale for selecting studies for replication.
Collapse
Affiliation(s)
- Merle-Marie Pittelkow
- Department of Psychometrics and Statistics, Rijksuniversiteit Groningen, Groningen, The Netherlands
| | - Sarahanne M. Field
- Centre of Science and Technology Studies, Leiden University, Leiden, the Netherlands
| | | | - Anna E. van’t Veer
- Methodology and Statistics Unit, Institute of Psychology, Leiden University, Leiden, the Netherlands
| | - Thomas Anderson
- Department of Psychology, University of Toronto, Toronto, Canada
| | - Scott N. Cole
- School of Education, Language and Psychology, York St John University, York, UK
| | - Tomáš Dominik
- Institute for Interdisciplinary Brain and Behavioral Sciences, Chapman University, Orange, CA, USA
| | | | - Sebahat Gok
- Program in Cognitive Science, Department of Instructional Systems Technology, Indiana University, Bloomington, IN, USA
| | - Tom Heyman
- Methodology and Statistics Unit, Institute of Psychology, Leiden University, Leiden, the Netherlands
| | - Marc Jekel
- Social Psychology, University of Cologne, Cologne, Germany
| | - Timothy J. Luke
- Department of Psychology, University of Gothenburg, Gothenburg, Sweden
| | - David B. Mitchell
- WellStar College of Health and Human Services, Kennesaw State University, Kennesaw, GA, USA
| | - Rik Peels
- Philosophy Department and the Faculty of Religion and Theology, Vrije Universiteit, Amsterdam, The Netherlands
| | - Rosina Pendrous
- Centre for Contextual Behavioural Science, School of Psychology, University of Chester, Chester, UK
- Institute of Applied Health Research, University of Birmingham, Birmingham, UK
| | - Samuel Sarrazin
- Maison de santé pluridisciplinaire Pasteur, Chevilly-Larue, France
| | - Jacob M. Schauer
- Department of Preventive Medicine - Division of Biostatistics, Feinberg School of Medicine, Northwestern University, Chicago, IL, USA
| | - Eva Specker
- Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria
| | - Ulrich S. Tran
- Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria
| | - Marek A. Vranka
- Faculty of Social Sciences, Charles University, Prague, Czech Republic
| | - Jelte M. Wicherts
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| | - Naoto Yoshimura
- Research Organization of Open Innovation and Collaboration, Ritsumeikan University, Osaka, Japan
- Research Fellow of the Japan Society for the Promotion of Science, Erasmus University Rotterdam, Rotterdam, The Netherlands
| | - Rolf A. Zwaan
- Department of Psychology, Education, and Child Studies, Erasmus University Rotterdam, Rotterdam, The Netherlands
| | - Don van Ravenzwaaij
- Department of Psychometrics and Statistics, Rijksuniversiteit Groningen, Groningen, The Netherlands
| |
Collapse
|
17
|
O’Connor DB, Abele AE, Baguley T, Daly M, Doherty N, Grange JA, Leman PJ, Pressman SD, Rossit S, Simms V. Improving psychological science: further thoughts, reflections and ways forward. COGENT PSYCHOLOGY 2022. [DOI: 10.1080/23311908.2022.2103248] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/16/2022] Open
Affiliation(s)
| | - Andrea E. Abele
- School of Psychology, University of Erlangen-Nuremberg, Erlangen, Germany
| | - Thom Baguley
- School of Psychology, Nottingham Trent University, Nottingham, UK
| | - Michael Daly
- Department of Psychology, Maynooth University, Ireland
| | - Nicola Doherty
- Paediatric Psychology Service, Western Health and Social Care Trust, Londonderry, UK
| | - James A. Grange
- School of Psychology, Keele University, Newcastle-under-Lyme, UK
| | - Patrick J. Leman
- Arts and Social Sciences, University of Waikato, Hamilton, New Zealand
| | - Sarah D. Pressman
- Department of Psychological Science, University of California, Irvine, California, USA
| | | | | |
Collapse
|
18
|
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. [The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsDeclaración PRISMA 2020: una guía actualizada para la publicación de revisiones sistemáticas]. Rev Panam Salud Publica 2022; 46:e112. [PMID: 36601438 PMCID: PMC9798848 DOI: 10.26633/rpsp.2022.112] [Citation(s) in RCA: 21] [Impact Index Per Article: 10.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/04/2021] [Indexed: 12/30/2022] Open
Abstract
The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, published in 2009, was designed to help systematic reviewers transparently report why the review was done, what the authors did, and what they found. Over the past decade, advances in systematic review methodology and terminology have necessitated an update to the guideline. The PRISMA 2020 statement replaces the 2009 statement and includes new reporting guidance that reflects advances in methods to identify, select, appraise, and synthesise studies. The structure and presentation of the items have been modified to facilitate implementation. In this article, we present the PRISMA 2020 27-item checklist, an expanded checklist that details reporting recommendations for each item, the PRISMA 2020 abstract checklist, and the revised flow diagrams for original and updated reviews.
Collapse
Affiliation(s)
- Matthew J. Page
- Monash UniversitySchool of Public Health and Preventive MedicineMelbourneAustráliaMonash University, School of Public Health and Preventive Medicine, Melbourne, Austrália
| | - Joanne E. McKenzie
- Monash UniversitySchool of Public Health and Preventive MedicineMelbourneAustráliaMonash University, School of Public Health and Preventive Medicine, Melbourne, Austrália
| | - Patrick M. Bossuyt
- University of AmsterdamAmsterdam University Medical CentresAmsterdãHolandaUniversity of Amsterdam, Amsterdam University Medical Centres, Amsterdã, Holanda.
| | - Isabelle Boutron
- Université de ParisCentre of Epidemiology and StatisticsParisFrançaUniversité de Paris, Centre of Epidemiology and Statistics, Paris, França
| | - Tammy C. Hoffmann
- Bond UniversityFaculty of Health Sciences and MedicineGold CoastAustráliaBond University, Faculty of Health Sciences and Medicine, Gold Coast, Austrália
| | - Cynthia D. Mulrow
- University of Texas Health Science Center at San AntonioSan AntonioTexasEstados UnidosUniversity of Texas Health Science Center at San Antonio, San Antonio, Texas, Estados Unidos
| | - Larissa Shamseer
- University of OttawaSchool of Epidemiology and Public HealthOttawaCanadáUniversity of Ottawa, School of Epidemiology and Public Health, Ottawa, Canadá
| | | | - Elie A. Akl
- American University of BeirutClinical Research InstituteBeiruteLíbanoAmerican University of Beirut, Clinical Research Institute, Beirute, Líbano
| | - Sue E. Brennan
- Monash UniversitySchool of Public Health and Preventive MedicineMelbourneAustráliaMonash University, School of Public Health and Preventive Medicine, Melbourne, Austrália
| | - Roger Chou
- Oregon Health & Science UniversityDepartment of Medical Informatics and Clinical EpidemiologyPortlandOregonEstados UnidosOregon Health & Science University, Department of Medical Informatics and Clinical Epidemiology, Portland, Oregon, Estados Unidos
| | - Julie Glanville
- University of YorkYork Health Economics ConsortiumYorkReino UnidoUniversity of York, York Health Economics Consortium, York, Reino Unido
| | - Jeremy M. Grimshaw
- Ottawa Hospital Research InstituteClinical Epidemiology ProgramOttawaCanadáOttawa Hospital Research Institute, Clinical Epidemiology Program, Ottawa, Canadá
| | - Asbjørn Hróbjartsson
- University of Southern DenmarkDepartment of Clinical ResearchOdenseDinamarcaUniversity of Southern Denmark, Department of Clinical Research, Odense, Dinamarca
| | - Manoj M. Lalu
- Ottawa HospitalDepartment of Anesthesiology and Pain MedicineOttawaCanadáOttawa Hospital, Department of Anesthesiology and Pain Medicine, Ottawa, Canadá
| | - Tianjing Li
- University of Colorado DenverSchool of MedicineDenverCloradoUnited StatesUniversity of Colorado Denver, School of Medicine, Denver, Colorado, United States
| | - Elizabeth W. Loder
- Harvard Medical SchoolBrigham and Women’s HospitalBostonMassachusettsEstados UnidosHarvard Medical School, Brigham and Women’s Hospital, Boston, Massachusetts, Estados Unidos
| | - Evan Mayo-Wilson
- Indiana University School of Public Health-BloomingtonDepartment of Epidemiology and BiostatisticsBloomingtonIndianaEstados UnidosIndiana University School of Public Health-Bloomington, Department of Epidemiology and Biostatistics, Bloomington, Indiana, Estados Unidos
| | - Steve McDonald
- Monash UniversitySchool of Public Health and Preventive MedicineMelbourneAustráliaMonash University, School of Public Health and Preventive Medicine, Melbourne, Austrália
| | - Luke A. McGuinness
- University of BristolBristol Medical SchoolBristolReino UnidoUniversity of Bristol, Bristol Medical School, Bristol, Reino Unido
| | - Lesley A. Stewart
- University of YorkCentre for Reviews and DisseminationYorkReino UnidoUniversity of York, Centre for Reviews and Dissemination, York, Reino Unido
| | - James Thomas
- University College LondonSocial Research InstituteLondonReino UnidoUniversity College London, Social Research Institute, London, Reino Unido
| | - Andrea C. Tricco
- University of TorontoInstitute of Health Management, Policy, and EvaluationTorontoCanadáUniversity of Toronto, Institute of Health Management, Policy, and Evaluation, Toronto, Canadá
| | - Vivian A. Welch
- Bruyère Research InstituteMethods CentreOttawaOntarioCanadáBruyère Research Institute, Methods Centre, Ottawa, Ontario, Canadá
| | - Penny Whiting
- University of BristolBristol Medical SchoolBristolReino UnidoUniversity of Bristol, Bristol Medical School, Bristol, Reino Unido
| | - David Moher
- Ottawa Hospital Research Institutecentre for JournalologyOttawaCanadáOttawa Hospital Research Institute, centre for Journalology, Ottawa, Canadá
| |
Collapse
|
19
|
Jacobucci R. A critique of using the labels confirmatory and exploratory in modern psychological research. Front Psychol 2022; 13:1020770. [PMID: 36582318 PMCID: PMC9792672 DOI: 10.3389/fpsyg.2022.1020770] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2022] [Accepted: 10/18/2022] [Indexed: 12/15/2022] Open
Abstract
Psychological science is experiencing a rise in the application of complex statistical models and, simultaneously, a renewed focus on applying research in a confirmatory manner. This presents a fundamental conflict for psychological researchers as more complex forms of modeling necessarily eschew as stringent of theoretical constraints. In this paper, I argue that this is less of a conflict, and more a result of a continued adherence to applying the overly simplistic labels of exploratory and confirmatory. These terms mask a distinction between exploratory/confirmatory research practices and modeling. Further, while many researchers recognize that this dichotomous distinction is better represented as a continuum, this only creates additional problems. Finally, I argue that while a focus on preregistration helps clarify the distinction, psychological research would be better off replacing the terms exploratory and confirmatory with additional levels of detail regarding the goals of the study, modeling details, and scientific method.
Collapse
|
20
|
Banks GC, Ross R, Toth AA, Tonidandel S, Mahdavi Goloujeh A, Dou W, Wesslen R. The triangulation of ethical leader signals using qualitative, experimental, and data science methods. THE LEADERSHIP QUARTERLY 2022. [DOI: 10.1016/j.leaqua.2022.101658] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
|
21
|
Votinov M, Knyazeva I, Habel U, Konrad K, Puiu AA. A Bayesian Modeling Approach to Examine the Role of Testosterone Administration on the Endowment Effect and Risk-Taking. Front Neurosci 2022; 16:858168. [PMID: 35937873 PMCID: PMC9347223 DOI: 10.3389/fnins.2022.858168] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2022] [Accepted: 06/22/2022] [Indexed: 11/17/2022] Open
Abstract
Financial risk-taking and loss aversion are multifaceted phenomena that are the focus of neuroscience, psychology, and economics research. A growing number of studies highlighted the role of hormones (particularly of testosterone) on socio-economic decision-making. However, the effects of testosterone on risk-taking under framing and consumer-based choices and preferences are inconclusive. We investigated the effects of 100 mg testosterone administration on aspects of decision-making within the Prospect Theory framework which is the most used descriptive model of decision-making under risk. We assessed risk-taking under framing and the endowment effect (effect of possession) using Bayesian modeling. Forty men participated in this double-blind placebo-controlled fully-randomized cross-over experiment and performed two tasks. One was a risk-taking task with binary choices under positive and negative framing associated with different probabilities. In the second task participants had to bid money for hedonic and utilitarian items. We observed a significant increase in serum testosterone concentrations after transdermal application. Compared to placebo, testosterone administration increased risk-taking under the positive framing (very large effect size) and decreased under the negative framing (moderate to small). The sensitivity to gain was positive in each framing. Our model showed that decision-making is jointly influenced by testosterone and the trade-off between gains and losses. However, while the endowment effect was more pronounced for hedonic than for utilitarian items, the effect was independent of testosterone. The findings provide novel information on the complex modulatory role of testosterone on risk-taking within the framework of prospect theory and shed light on mechanisms of behavioral economic biases. The proposed models of effects of individual differences in testosterone on risk-taking could be used as predictive models for reference-depended behavior under positive and negative framing with low and high probabilities.
Collapse
Affiliation(s)
- Mikhail Votinov
- Research Center Jülich, Institute of Neuroscience and Medicine, JARA-Institute Brain Structure Function Relationship (INM 10), Julich, Germany
- Department of Psychiatry, Psychotherapy and Psychosomatics, Faculty of Medicine, RWTH Aachen University, Aachen, Germany
- *Correspondence: Mikhail Votinov
| | - Irina Knyazeva
- N.P. Bechtereva Institute of Human Brain, Russian Academy of Science, Saint Petersburg, Russia
- Saint Petersburg State University, Saint Petersburg, Russia
| | - Ute Habel
- Research Center Jülich, Institute of Neuroscience and Medicine, JARA-Institute Brain Structure Function Relationship (INM 10), Julich, Germany
- Department of Psychiatry, Psychotherapy and Psychosomatics, Faculty of Medicine, RWTH Aachen University, Aachen, Germany
| | - Kerstin Konrad
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Faculty of Medicine, RWTH Aachen University, Aachen, Germany
| | - Andrei A. Puiu
- Department of Psychiatry, Psychotherapy and Psychosomatics, Faculty of Medicine, RWTH Aachen University, Aachen, Germany
- Andrei A. Puiu
| |
Collapse
|
22
|
Sarafoglou A, Kovacs M, Bakos B, Wagenmakers EJ, Aczel B. A survey on how preregistration affects the research workflow: better science but more work. ROYAL SOCIETY OPEN SCIENCE 2022; 9:211997. [PMID: 35814910 PMCID: PMC9257590 DOI: 10.1098/rsos.211997] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/20/2021] [Accepted: 06/06/2022] [Indexed: 05/20/2023]
Abstract
The preregistration of research protocols and analysis plans is a main reform innovation to counteract confirmation bias in the social and behavioural sciences. While theoretical reasons to preregister are frequently discussed in the literature, the individually experienced advantages and disadvantages of this method remain largely unexplored. The goal of this exploratory study was to identify the perceived benefits and challenges of preregistration from the researcher's perspective. To this end, we surveyed 355 researchers, 299 of whom had used preregistration in their own work. The researchers indicated the experienced or expected effects of preregistration on their workflow. The results show that experiences and expectations are mostly positive. Researchers in our sample believe that implementing preregistration improves or is likely to improve the quality of their projects. Criticism of preregistration is primarily related to the increase in work-related stress and the overall duration of the project. While the benefits outweighed the challenges for the majority of researchers with preregistration experience, this was not the case for the majority of researchers without preregistration experience. The experienced advantages and disadvantages identified in our survey could inform future efforts to improve preregistration and thus help the methodology gain greater acceptance in the scientific community.
Collapse
Affiliation(s)
| | - Marton Kovacs
- Doctoral School of Psychology, ELTE Eotvos Lorand University, Budapest, Hungary
- Institute of Psychology, ELTE Eotvos Lorand University, Budapest, Hungary
| | - Bence Bakos
- Institute of Psychology, ELTE Eotvos Lorand University, Budapest, Hungary
| | | | - Balazs Aczel
- Institute of Psychology, ELTE Eotvos Lorand University, Budapest, Hungary
| |
Collapse
|
23
|
On Bayesian integration in sensorimotor learning: Another look at Kording and Wolpert (2004). Cortex 2022; 153:87-96. [DOI: 10.1016/j.cortex.2022.03.021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2022] [Accepted: 03/14/2022] [Indexed: 11/20/2022]
|
24
|
Kowalczyk OS, Lautarescu A, Blok E, Dall'Aglio L, Westwood SJ. What senior academics can do to support reproducible and open research: a short, three-step guide. BMC Res Notes 2022; 15:116. [PMID: 35317865 PMCID: PMC8938725 DOI: 10.1186/s13104-022-05999-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/24/2021] [Accepted: 03/09/2022] [Indexed: 01/31/2023] Open
Abstract
Increasingly, policies are being introduced to reward and recognise open research practices, while the adoption of such practices into research routines is being facilitated by many grassroots initiatives. However, despite this widespread endorsement and support, as well as various efforts led by early career researchers, open research is yet to be widely adopted. For open research to become the norm, initiatives should engage academics from all career stages, particularly senior academics (namely senior lecturers, readers, professors) given their routine involvement in determining the quality of research. Senior academics, however, face unique challenges in implementing policy changes and supporting grassroots initiatives. Given that-like all researchers-senior academics are motivated by self-interest, this paper lays out three feasible steps that senior academics can take to improve the quality and productivity of their research, that also serve to engender open research. These steps include changing (a) hiring criteria, (b) how scholarly outputs are credited, and (c) how we fund and publish in line with open research principles. The guidance we provide is accompanied by material for further reading.
Collapse
Affiliation(s)
- Olivia S Kowalczyk
- Department of Neuroimaging, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
| | - Alexandra Lautarescu
- Forensic and Neurodevelopmental Sciences, Institute of Psychiatry, Psychology, Neuroscience, King's College London, London, UK
- Department of Perinatal Imaging and Health, Centre for the Developing Brain, School of Biomedical Imaging and Medical Sciences, King's College London, London, UK
| | - Elisabet Blok
- Department of Child and Adolescent Psychiatry/Psychology, Erasmus MC-Sophia Children's Hospital, University Medical Centre Rotterdam, Rotterdam, The Netherlands
- The Generation R Study Group, Erasmus MC, University Medical Centre Rotterdam, Rotterdam, The Netherlands
| | - Lorenza Dall'Aglio
- Department of Child and Adolescent Psychiatry/Psychology, Erasmus MC-Sophia Children's Hospital, University Medical Centre Rotterdam, Rotterdam, The Netherlands
- The Generation R Study Group, Erasmus MC, University Medical Centre Rotterdam, Rotterdam, The Netherlands
| | - Samuel J Westwood
- Institute of Psychiatry, Psychology, Neuroscience, King's College London, London, UK.
- Department of Psychology, School of Social Science, University of Westminster, 115 New Cavendish Street, London, W1W 6UW, UK.
| |
Collapse
|
25
|
Sarafoglou A, van der Heijden A, Draws T, Cornelisse J, Wagenmakers EJ, Marsman M. Combine Statistical Thinking With Open Scientific Practice: A Protocol of a Bayesian Research Project. PSYCHOLOGY LEARNING AND TEACHING-PLAT 2022. [DOI: 10.1177/14757257221077307] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Current developments in the statistics community suggest that modern statistics education should be structured holistically, that is, by allowing students to work with real data and to answer concrete statistical questions, but also by educating them about alternative frameworks, such as Bayesian inference. In this article, we describe how we incorporated such a holistic structure in a Bayesian research project on ordered binomial probabilities. The project was conducted with a group of three undergraduate psychology students who had basic knowledge of Bayesian statistics and programming, but lacked formal mathematical training. The research project aimed to (1) convey the basic mathematical concepts of Bayesian inference; (2) have students experience the entire empirical cycle including collection, analysis, and interpretation of data and (3) teach students open science practices.
Collapse
Affiliation(s)
| | - Anna van der Heijden
- Department of Psychology, University of Amsterdam, The Netherlands
- Currently at De Groene Lobby, The Netherlands
| | - Tim Draws
- Faculty of Engineering, Mathematics and Computer Science, Delft University of Technology, The Netherlands
| | - Joran Cornelisse
- Department of Psychology, University of Amsterdam, The Netherlands
- Currently at SocialDatabase, The Netherlands
| | | | - Maarten Marsman
- Department of Psychology, University of Amsterdam, The Netherlands
| |
Collapse
|
26
|
Leach S, Piazza J, Loughnan S, Sutton RM, Kapantai I, Dhont K, Douglas KM. Unpalatable truths: Commitment to eating meat is associated with strategic ignorance of food-animal minds. Appetite 2022; 171:105935. [PMID: 35045324 DOI: 10.1016/j.appet.2022.105935] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2021] [Revised: 12/29/2021] [Accepted: 01/12/2022] [Indexed: 11/16/2022]
Abstract
Animal minds are of central importance to debates about their rights and welfare. Remaining ignorant of evidence that animals have minds is therefore likely to facilitate their mistreatment. Studying samples of adults and students from the UK and US we found that, consistent with motivational perspectives on meat consumption, those who were more (vs. less) committed to eating meat were more motivated to avoid exposure to information about food-animals' sentience (Studies 1), showed less interest in exposure to articles about intelligent food animals (Studies 2a and 2b), and were quicker to terminate exposure to internet pop-ups containing information about food-animals' minds (Studies 3a and 3b). At the same time, those who were more (vs. less) committed to eating meat approached information about companion-animals' minds (Studies 2a-3b) and unintelligent food animals (Studies 2a and 2b) in largely the same ways. The findings demonstrate that, within the UK and US, the desire to eat meat is associated with strategies to avoid information that is likely to challenge meat consumption.
Collapse
Affiliation(s)
| | - Jared Piazza
- Department of Psychology, Lancaster University, UK
| | - Steve Loughnan
- School of Philosophy, Psychology, and Language Sciences, University of Edinburgh, UK
| | | | - Ioanna Kapantai
- College of Life and Environmental Sciences, University of Exeter, UK
| | | | | |
Collapse
|
27
|
Szaszi B, Hajdu N, Szecsi P, Tipton E, Aczel B. A machine learning analysis of the relationship of demographics and social gathering attendance from 41 countries during pandemic. Sci Rep 2022; 12:724. [PMID: 35031631 PMCID: PMC8760248 DOI: 10.1038/s41598-021-04305-5] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2021] [Accepted: 12/07/2021] [Indexed: 12/17/2022] Open
Abstract
Knowing who to target with certain messages is the prerequisite of efficient public health campaigns during pandemics. Using the COVID-19 pandemic situation, we explored which facets of the society-defined by age, gender, income, and education levels-are the most likely to visit social gatherings and aggravate the spread of a disease. Analyzing the reported behavior of 87,169 individuals from 41 countries, we found that in the majority of the countries, the proportion of social gathering-goers was higher in male than female, younger than older, lower-educated than higher educated, and low-income than high-income subgroups of the populations. However, the data showed noteworthy heterogeneity between the countries warranting against generalizing from one country to another. The analysis also revealed that relative to other demographic factors, income was the strongest predictor of avoidance of social gatherings followed by age, education, and gender. Although the observed strength of these associations was relatively small, we argue that incorporating demographic-based segmentation into public health campaigns can increase the efficiency of campaigns with an important caveat: the exploration of these associations needs to be done on a country level before using the information to target populations in behavior change interventions.
Collapse
Affiliation(s)
- Barnabas Szaszi
- Institue of Psychology, ELTE Eötvös Loránd University, Budapest, Hungary.
| | - Nandor Hajdu
- Institue of Psychology, ELTE Eötvös Loránd University, Budapest, Hungary
- Doctoral School of Psychology, Institute of Psychology, Eötvös Loránd University, Budapest, Hungary
| | - Peter Szecsi
- Institue of Psychology, ELTE Eötvös Loránd University, Budapest, Hungary
- Doctoral School of Psychology, Institute of Psychology, Eötvös Loránd University, Budapest, Hungary
| | | | - Balazs Aczel
- Institue of Psychology, ELTE Eötvös Loránd University, Budapest, Hungary
| |
Collapse
|
28
|
Jarke H, Jakob L, Bojanić L, Garcia-Garzon E, Mareva S, Mutak A, Gjorgjiovska J. Registered report: How open do you want your science? An international investigation into knowledge and attitudes of psychology students. PLoS One 2022; 17:e0261260. [PMID: 35226677 PMCID: PMC8884512 DOI: 10.1371/journal.pone.0261260] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2021] [Accepted: 11/27/2021] [Indexed: 12/04/2022] Open
Abstract
The use of Open Science practices is often proposed as a way to improve research practice, especially in psychology. Open Science can increase transparency and therefore reduce questionable research practices, making research more accessible to students, scholars, policy makers, and the public. However, little is known about how widespread Open Science practices are taught and how students are educated about these practices. In addition, it remains unknown how informing students about Open Science actually impacts their understanding and adoption of such practices. This registered report proposes the validation of a questionnaire. The aim is to survey how much psychology students know about Open Science and to assess whether knowledge of and exposure to Open Science in general-be it through university curricula or social media-influences attitudes towards the concept and intentions to implement relevant practices.
Collapse
Affiliation(s)
- Hannes Jarke
- Junior Researcher Programme, Cambridge, United Kingdom
- Centre for Business Research, Judge Business School, University of Cambridge, United Kingdom
- * E-mail:
| | - Lea Jakob
- Junior Researcher Programme, Cambridge, United Kingdom
- National Institute of Mental Health, Klecany, Czech Republic
- 3rd Faculty of Medicine, Charles University, Czech Republic
| | - Lana Bojanić
- Junior Researcher Programme, Cambridge, United Kingdom
- Division of Psychology and Mental Health, Faculty of Biology, Medicine, and Health, University of Manchester, Manchester, United Kingdom
| | - Eduardo Garcia-Garzon
- Junior Researcher Programme, Cambridge, United Kingdom
- Facultad de Salud, Universidad Camilo José Cela, Madrid, Spain
| | - Silvana Mareva
- Junior Researcher Programme, Cambridge, United Kingdom
- MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, United Kingdom
| | - Augustin Mutak
- Junior Researcher Programme, Cambridge, United Kingdom
- Methods and Evaluation/Quality Assurance, Freie Universität Berlin, Berlin, Germany
| | | |
Collapse
|
29
|
Hagger MS. Developing an open science 'mindset'. Health Psychol Behav Med 2021; 10:1-21. [PMID: 34993003 PMCID: PMC8725925 DOI: 10.1080/21642850.2021.2012474] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2021] [Accepted: 11/24/2021] [Indexed: 11/21/2022] Open
Abstract
BACKGROUND Identification of widespread biases present in reported research findings in many scientific disciplines, including psychology, such as failures to replicate and the likely extensive application of questionable research practices, has raised serious concerns over the reliability and trustworthiness of scientific research. This has led to the development of, and advocacy for, 'open science' practices, including data, materials, analysis, and output sharing, pre-registration of study predictions and analysis plans, and increased access to published research findings. Implementation of such practices has been enthusiastic in some quarters, but literacy in, and adoption of, these practices has lagged behind among many researchers in the scientific community. ADVANCES In the current article I propose that researchers adopt an open science 'mindset', a comprehensive approach to open science predicated on researchers' operating under the basic assumption that, wherever possible, open science practices will be a central component of all steps of their research projects. The primary, defining feature of the mindset is a commitment to open science principles in all research projects from inception to dissemination. Other features of the mindset include the assumption that all components of research projects (e.g. pre-registered hypotheses, protocols, materials, analysis plans, data, and output) will be accessible broadly; pro-active selection of open fora to disseminate research components and findings; open and transparent dissemination of reports of the research findings in advance of, and after, formal publication; and active promotion of open science practices through education, modeling, and advocacy. CONCLUSION The open science mindset is a 'farm to fork' approach to open science aimed at promoting comprehensive quality in application of open science, and widening participation in open science practices so that they become the norm in research in health psychology and behavioral medicine going forward.
Collapse
Affiliation(s)
- Martin S. Hagger
- Department of Psychological Sciences and Health Sciences Research Institute, University of California, Merced, CA, USA
- Faculty of Sport and Health Sciences, University of Jyväskylä, Jyväskylä, Finland
- School of Applied Psychology, Griffith University, Mt. Gravatt, Australia
| |
Collapse
|
30
|
Aczel B, Szaszi B, Nilsonne G, van den Akker OR, Albers CJ, van Assen MALM, Bastiaansen JA, Benjamin D, Boehm U, Botvinik-Nezer R, Bringmann LF, Busch NA, Caruyer E, Cataldo AM, Cowan N, Delios A, van Dongen NNN, Donkin C, van Doorn JB, Dreber A, Dutilh G, Egan GF, Gernsbacher MA, Hoekstra R, Hoffmann S, Holzmeister F, Huber J, Johannesson M, Jonas KJ, Kindel AT, Kirchler M, Kunkels YK, Lindsay DS, Mangin JF, Matzke D, Munafò MR, Newell BR, Nosek BA, Poldrack RA, van Ravenzwaaij D, Rieskamp J, Salganik MJ, Sarafoglou A, Schonberg T, Schweinsberg M, Shanks D, Silberzahn R, Simons DJ, Spellman BA, St-Jean S, Starns JJ, Uhlmann EL, Wicherts J, Wagenmakers EJ. Consensus-based guidance for conducting and reporting multi-analyst studies. eLife 2021; 10:e72185. [PMID: 34751133 PMCID: PMC8626083 DOI: 10.7554/elife.72185] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2021] [Accepted: 11/07/2021] [Indexed: 11/13/2022] Open
Abstract
Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.
Collapse
Affiliation(s)
| | | | - Gustav Nilsonne
- Karolinska InstitutetStockholmSweden
- Stockholm UniversityStockholmSweden
| | | | | | | | - Jojanneke A Bastiaansen
- University Medical Center Groningen, University of GroningenGroningenNetherlands
- Friesland Mental Health Care ServicesLeeuwardenNetherlands
| | - Daniel Benjamin
- University of California Los AngelesLos AngelesUnited States
- National Bureau of Economic ResearchCambridgeUnited States
| | - Udo Boehm
- University of AmsterdamAmsterdamNetherlands
| | | | | | | | | | - Andrea M Cataldo
- McLean HospitalBelmontUnited States
- Harvard Medical SchoolBostonUnited States
| | | | | | | | | | | | - Anna Dreber
- Stockholm School of EconomicsStockholmSweden
- University of InnsbruckInnsbruckAustria
| | | | | | | | | | | | | | | | | | | | | | | | - Yoram K Kunkels
- University Medical Center Groningen, University of GroningenGroningenNetherlands
| | | | | | | | | | | | - Brian A Nosek
- Center for Open ScienceCharlottesvilleUnited States
- University of VirginiaCharlottesvilleUnited States
| | | | | | | | | | | | | | | | | | | | | | | | - Samuel St-Jean
- University of AlbertaEdmontonCanada
- Lund UniversityLundUnited States
| | | | | | | | | |
Collapse
|
31
|
Seven steps toward more transparency in statistical practice. Nat Hum Behav 2021; 5:1473-1480. [PMID: 34764461 DOI: 10.1038/s41562-021-01211-8] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2021] [Accepted: 09/06/2021] [Indexed: 01/03/2023]
Abstract
We argue that statistical practice in the social and behavioural sciences benefits from transparency, a fair acknowledgement of uncertainty and openness to alternative interpretations. Here, to promote such a practice, we recommend seven concrete statistical procedures: (1) visualizing data; (2) quantifying inferential uncertainty; (3) assessing data preprocessing choices; (4) reporting multiple models; (5) involving multiple analysts; (6) interpreting results modestly; and (7) sharing data and code. We discuss their benefits and limitations, and provide guidelines for adoption. Each of the seven procedures finds inspiration in Merton's ethos of science as reflected in the norms of communalism, universalism, disinterestedness and organized scepticism. We believe that these ethical considerations-as well as their statistical consequences-establish common ground among data analysts, despite continuing disagreements about the foundations of statistical inference.
Collapse
|
32
|
Shanks DR, Vadillo MA. Publication bias and low power in field studies on goal priming. ROYAL SOCIETY OPEN SCIENCE 2021; 8:210544. [PMID: 34667618 PMCID: PMC8522447 DOI: 10.1098/rsos.210544] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/12/2021] [Accepted: 09/16/2021] [Indexed: 05/09/2023]
Abstract
Research on goal priming asks whether the subtle activation of an achievement goal can improve task performance. Studies in this domain employ a range of priming methods, such as surreptitiously displaying a photograph of an athlete winning a race, and a range of dependent variables including measures of creativity and workplace performance. Chen, Latham, Piccolo and Itzchakov (Chen et al. 2021 J. Appl. Psychol. 70, 216-253) recently undertook a meta-analysis of this research and reported positive overall effects in both laboratory and field studies, with field studies yielding a moderate-to-large effect that was significantly larger than that obtained in laboratory experiments. We highlight a number of issues with Chen et al.'s selection of field studies and then report a new meta-analysis (k = 13, N = 683) that corrects these. The new meta-analysis reveals suggestive evidence of publication bias and low power in goal priming field studies. We conclude that the available evidence falls short of demonstrating goal priming effects in the workplace, and offer proposals for how future research can provide stronger tests.
Collapse
Affiliation(s)
- David R. Shanks
- Division of Psychology and Language Sciences, University College London, 26 Bedford Way, London WC1H 0AP, UK
| | - Miguel A. Vadillo
- Departamento de Psicología Básica, Universidad Autónoma de Madrid, Spain
| |
Collapse
|
33
|
O'Dea RE, Lagisz M, Jennions MD, Koricheva J, Noble DW, Parker TH, Gurevitch J, Page MJ, Stewart G, Moher D, Nakagawa S. Preferred reporting items for systematic reviews and meta-analyses in ecology and evolutionary biology: a PRISMA extension. Biol Rev Camb Philos Soc 2021; 96:1695-1722. [PMID: 33960637 PMCID: PMC8518748 DOI: 10.1111/brv.12721] [Citation(s) in RCA: 129] [Impact Index Per Article: 43.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2020] [Revised: 04/05/2021] [Accepted: 04/08/2021] [Indexed: 12/29/2022]
Abstract
Since the early 1990s, ecologists and evolutionary biologists have aggregated primary research using meta-analytic methods to understand ecological and evolutionary phenomena. Meta-analyses can resolve long-standing disputes, dispel spurious claims, and generate new research questions. At their worst, however, meta-analysis publications are wolves in sheep's clothing: subjective with biased conclusions, hidden under coats of objective authority. Conclusions can be rendered unreliable by inappropriate statistical methods, problems with the methods used to select primary research, or problems within the primary research itself. Because of these risks, meta-analyses are increasingly conducted as part of systematic reviews, which use structured, transparent, and reproducible methods to collate and summarise evidence. For readers to determine whether the conclusions from a systematic review or meta-analysis should be trusted - and to be able to build upon the review - authors need to report what they did, why they did it, and what they found. Complete, transparent, and reproducible reporting is measured by 'reporting quality'. To assess perceptions and standards of reporting quality of systematic reviews and meta-analyses published in ecology and evolutionary biology, we surveyed 208 researchers with relevant experience (as authors, reviewers, or editors), and conducted detailed evaluations of 102 systematic review and meta-analysis papers published between 2010 and 2019. Reporting quality was far below optimal and approximately normally distributed. Measured reporting quality was lower than what the community perceived, particularly for the systematic review methods required to measure trustworthiness. The minority of assessed papers that referenced a guideline (~16%) showed substantially higher reporting quality than average, and surveyed researchers showed interest in using a reporting guideline to improve reporting quality. The leading guideline for improving reporting quality of systematic reviews is the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement. Here we unveil an extension of PRISMA to serve the meta-analysis community in ecology and evolutionary biology: PRISMA-EcoEvo (version 1.0). PRISMA-EcoEvo is a checklist of 27 main items that, when applicable, should be reported in systematic review and meta-analysis publications summarising primary research in ecology and evolutionary biology. In this explanation and elaboration document, we provide guidance for authors, reviewers, and editors, with explanations for each item on the checklist, including supplementary examples from published papers. Authors can consult this PRISMA-EcoEvo guideline both in the planning and writing stages of a systematic review and meta-analysis, to increase reporting quality of submitted manuscripts. Reviewers and editors can use the checklist to assess reporting quality in the manuscripts they review. Overall, PRISMA-EcoEvo is a resource for the ecology and evolutionary biology community to facilitate transparent and comprehensively reported systematic reviews and meta-analyses.
Collapse
Affiliation(s)
- Rose E. O'Dea
- Evolution & Ecology Research Centre and School of Biological and Environmental SciencesUniversity of New South WalesSydneyNSW2052Australia
| | - Malgorzata Lagisz
- Evolution & Ecology Research Centre and School of Biological and Environmental SciencesUniversity of New South WalesSydneyNSW2052Australia
| | - Michael D. Jennions
- Research School of BiologyAustralian National University46 Sullivans Creek RoadCanberra2600Australia
| | - Julia Koricheva
- Department of Biological SciencesRoyal Holloway University of LondonEghamSurreyTW20 0EXU.K.
| | - Daniel W.A. Noble
- Evolution & Ecology Research Centre and School of Biological and Environmental SciencesUniversity of New South WalesSydneyNSW2052Australia
- Research School of BiologyAustralian National University46 Sullivans Creek RoadCanberra2600Australia
| | | | - Jessica Gurevitch
- Department of Ecology and EvolutionStony Brook UniversityStony BrookNY11794‐5245U.S.A.
| | - Matthew J. Page
- School of Public Health and Preventative MedicineMonash UniversityMelbourneVIC3004Australia
| | - Gavin Stewart
- School of Natural and Environmental SciencesNewcastle UniversityNewcastle upon TyneNE1 7RUU.K.
| | - David Moher
- Centre for Journalology, Clinical Epidemiology ProgramOttawa Hospital Research InstituteGeneral Campus, 501 Smyth Road, Room L1288OttawaONK1H 8L6Canada
| | - Shinichi Nakagawa
- Evolution & Ecology Research Centre and School of Biological and Environmental SciencesUniversity of New South WalesSydneyNSW2052Australia
| |
Collapse
|
34
|
van Dongen N, Sikorski M. Objectivity for the research worker. EUROPEAN JOURNAL FOR PHILOSOPHY OF SCIENCE 2021; 11:93. [PMID: 34721744 PMCID: PMC8550135 DOI: 10.1007/s13194-021-00400-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/06/2020] [Accepted: 07/12/2021] [Indexed: 06/13/2023]
Abstract
In the last decade, many problematic cases of scientific conduct have been diagnosed; some of which involve outright fraud (e.g., Stapel, 2012) others are more subtle (e.g., supposed evidence of extrasensory perception; Bem, 2011). These and similar problems can be interpreted as caused by lack of scientific objectivity. The current philosophical theories of objectivity do not provide scientists with conceptualizations that can be effectively put into practice in remedying these issues. We propose a novel way of thinking about objectivity for individual scientists; a negative and dynamic approach.We provide a philosophical conceptualization of objectivity that is informed by empirical research. In particular, it is our intention to take the first steps in providing an empirically and methodologically informed inventory of factors that impair the scientific practice. The inventory will be compiled into a negative conceptualization (i.e., what is not objective), which could in principle be used by individual scientists to assess (deviations from) objectivity of scientific practice. We propose a preliminary outline of a usable and testable instrument for indicating the objectivity of scientific practice.
Collapse
Affiliation(s)
| | - Michał Sikorski
- University of Gdańsk, Gdańsk, Poland
- Warsaw University of Technology, Warsaw, Poland
| |
Collapse
|
35
|
Lecours A, Beaulieu AA, Poulin V, Nastasia I, St-Hilaire F. Development of a Questionnaire Measuring Preventive Behaviors at Work. JOURNAL OF OCCUPATIONAL REHABILITATION 2021; 31:570-580. [PMID: 33398507 DOI: 10.1007/s10926-020-09955-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 12/17/2020] [Indexed: 06/12/2023]
Abstract
Purpose This study aimed to develop a questionnaire measuring preventive behaviors at work. Methods A three-step design, including qualitative and quantitative methods, was followed: (1) item generation, (2) experts' validation of content, and (3) pretesting. Results For step 1, 49 relevant existing scales were reviewed, and a pool of 172 items was generated. Redundant items were deleted (n = 48), and unclear items were reworded (n = 27). For step 2, 14 experts (five occupational therapists, four researchers, and five workers) assessed the representativeness, relevance, and clarity of each item through content validity indices (CVIs). An average overall CVI of 0.97 was obtained, and 87.5% of the experts stated that the questionnaire was comprehensive. During this step, 63 items were deleted, and 35 were modified. For step 3, the tool was pretested in the clinical settings of four dyads (occupational therapist-worker). The thematic analysis of interview content allowed several changes to be made to the questionnaire, including the addition of information and format changes. Conclusions Overall, this three-step study led to the construction of a 61-item French questionnaire entitled the Échelle de fréquence des comportements préventifs au travail [Frequency Scale of Preventive Behaviors at Work]. In rehabilitation settings, this tool could be useful to support professionals in enabling workers to adopt preventive behaviors, thereby fostering a healthy, sustainable return to work after a disability period. However, further metrological property assessment is required. A validating study using a large pool of workers is ongoing.
Collapse
Affiliation(s)
- Alexandra Lecours
- Département de réadaptation, Université Laval, Pavillon Ferdinand-Vandry 1050, Avenue de la Médecine, Québec, QC, G1V 0A6, Canada.
- Center for Interdisciplinary Research in Rehabilitation and Social Integration, Québec, Canada.
- Équipe sur les organisations en santé, École de gestion, Université de Sherbrooke, Sherbrooke, Canada.
| | - Alex-Anne Beaulieu
- Center for Interdisciplinary Research in Rehabilitation and Social Integration, Québec, Canada
| | - Valérie Poulin
- Center for Interdisciplinary Research in Rehabilitation and Social Integration, Québec, Canada
- Département d'ergothérapie, Université du Québec à Trois-Rivières, Trois-Rivières, Canada
| | - Iuliana Nastasia
- Institut de Recherche Robert-Sauvé en Santé et Sécurité du Travail, Montréal, Canada
| | - France St-Hilaire
- Équipe sur les organisations en santé, École de gestion, Université de Sherbrooke, Sherbrooke, Canada
- Département de management et gestion des ressources humaines, École de gestion, Université de Sherbrooke, Sherbrooke, Canada
| |
Collapse
|
36
|
Declaración PRISMA 2020: una guía actualizada para la publicación de revisiones sistemáticas. Rev Esp Cardiol 2021. [DOI: 10.1016/j.recesp.2021.06.016] [Citation(s) in RCA: 76] [Impact Index Per Article: 25.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/26/2022]
|
37
|
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. REVISTA ESPANOLA DE CARDIOLOGIA (ENGLISH ED.) 2021; 74:790-799. [PMID: 34446261 DOI: 10.1016/j.rec.2021.07.010] [Citation(s) in RCA: 159] [Impact Index Per Article: 53.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/21/2021] [Accepted: 06/24/2021] [Indexed: 12/20/2022]
Abstract
The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, published in 2009, was designed to help systematic reviewers transparently report why the review was done, what the authors did, and what they found. Over the past decade, advances in systematic review methodology and terminology have necessitated an update to the guideline. The PRISMA 2020 statement replaces the 2009 statement and includes new reporting guidance that reflects advances in methods to identify, select, appraise, and synthesise studies. The structure and presentation of the items have been modified to facilitate implementation. In this article, we present the PRISMA 2020 27-item checklist, an expanded checklist that details reporting recommendations for each item, the PRISMA 2020 abstract checklist, and the revised flow diagrams for original and updated reviews. Full English text available from:www.revespcardiol.org/en.
Collapse
Affiliation(s)
- Matthew J Page
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia.
| | - Joanne E McKenzie
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Patrick M Bossuyt
- Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Amsterdam University Medical Centres, University of Amsterdam, Amsterdam, Países Bajos
| | - Isabelle Boutron
- Université de Paris, Centre of Epidemiology and Statistics (CRESS), Inserm, F 75004 Paris, Francia
| | - Tammy C Hoffmann
- Institute for Evidence-Based Healthcare, Faculty of Health Sciences and Medicine, Bond University, Gold Coast, Australia
| | - Cynthia D Mulrow
- University of Texas Health Science Center at San Antonio, San Antonio, Texas, Estados Unidos; Annals of Internal Medicine
| | - Larissa Shamseer
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, Toronto, Canadá; School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Canada
| | | | - Elie A Akl
- Clinical Research Institute, American University of Beirut, Beirut, Líbano; Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canadá
| | - Sue E Brennan
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Roger Chou
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, Oregón, Estados Unidos
| | - Julie Glanville
- York Health Economics Consortium (YHEC Ltd), University of York, York, Reino Unido
| | - Jeremy M Grimshaw
- Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canadá; School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canadá; Department of Medicine, University of Ottawa, Ottawa, Canadá
| | - Asbjørn Hróbjartsson
- Centre for Evidence-Based Medicine Odense, Odense University Hospital, Odense, Dinamarca; Department of Clinical Research, University of Southern Denmark, Odense, Dinamarca; Open Patient data Explorative Network, Odense University Hospital, Odense, Dinamarca
| | - Manoj M Lalu
- Department of Anesthesiology and Pain Medicine, The Ottawa Hospital, Ottawa, Canadá; Clinical Epidemiology Program, Blueprint Translational Research Group, Ottawa Hospital Research Institute, Ottawa, Canadá; Regenerative Medicine Program, Ottawa Hospital Research Institute, Ottawa, Canadá
| | - Tianjing Li
- Department of Ophthalmology, School of Medicine, University of Colorado Denver, Denver, Colorado, Estados Unidos; Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, Estados Unidos
| | - Elizabeth W Loder
- Division of Headache, Department of Neurology, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, Estados Unidos; Head of Research, The BMJ, Londres, Reino Unido
| | - Evan Mayo-Wilson
- Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, Indiana, Estados Unidos
| | - Steve McDonald
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Luke A McGuinness
- Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, Reino Unido
| | - Lesley A Stewart
- Centre for Reviews and Dissemination, University of York, York, Reino Unido
| | - James Thomas
- EPPI-Centre, UCL Social Research Institute, University College London, Londres, Reino Unido
| | - Andrea C Tricco
- Li Ka Shing Knowledge Institute of St. Michael's Hospital, Unity Health Toronto, Toronto, Canadá; Epidemiology Division of the Dalla Lana School of Public Health and the Institute of Health Management, Policy, and Evaluation, University of Toronto, Toronto, Canadá; Queen's Collaboration for Health Care Quality Joanna Briggs Institute Centre of Excellence, Queen's University, Kingston, Canadá
| | - Vivian A Welch
- Methods Centre, Bruyère Research Institute, Ottawa, Ontario, Canadá; School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Canadá
| | - Penny Whiting
- Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, Reino Unido
| | - David Moher
- Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canadá; School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Canadá
| |
Collapse
|
38
|
Ernst BA, Banks GC, Loignon AC, Frear KA, Williams CE, Arciniega LM, Gupta RK, Kodydek G, Subramanian D. Virtual charismatic leadership and signaling theory: A prospective meta-analysis in five countries. THE LEADERSHIP QUARTERLY 2021. [DOI: 10.1016/j.leaqua.2021.101541] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
|
39
|
Mertens A, Mertens UK, Lerche V. On the difficulty to think in ratios: a methodological bias in Stevens' magnitude estimation procedure. Atten Percept Psychophys 2021; 83:2347-2365. [PMID: 33791941 PMCID: PMC8213559 DOI: 10.3758/s13414-021-02266-5] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/24/2021] [Indexed: 11/11/2022]
Abstract
In the field of new psychophysics, the magnitude estimation procedure is one of the most frequently used methods. It requires participants to assess the intensity of a stimulus in relation to a reference. In three studies, we examined whether difficulties of thinking in ratios influence participants' intensity perceptions. In Study 1, a standard magnitude estimation procedure was compared to an adapted procedure in which the numerical response dimension was reversed so that smaller (larger) numbers indicated brighter (darker) stimuli. In Study 2, participants first had to indicate whether a stimulus was brighter or darker compared to the reference, and only afterwards they estimated the magnitude of this difference, always using ratings above the reference to indicate their perception. In Study 3, we applied the same procedure as in Study 2 to a different physical dimension (red saturation). Results from Study 1 (N = 20) showed that participants in the reversal condition used more (less) extreme ratings for brighter (darker) stimuli compared to the standard condition. Data from the unidirectional method applied in Study 2 (N = 34) suggested a linear psychophysical function for brightness perception. Similar results were found for red saturation in Study 3 (N = 36) with a less curved power function describing the association between objective red saturation and perceived redness perception. We conclude that the typical power functions that emerge when using a standard magnitude estimation procedure might be biased due to difficulties experienced by participants to think in ratios.
Collapse
Affiliation(s)
- Alica Mertens
- Institute of Psychology, Heidelberg University, Hauptstrasse 47-51, D-69117, Heidelberg, Germany.
| | - Ulf K Mertens
- Institute of Psychology, Heidelberg University, Hauptstrasse 47-51, D-69117, Heidelberg, Germany
| | - Veronika Lerche
- Institute of Psychology, Heidelberg University, Hauptstrasse 47-51, D-69117, Heidelberg, Germany
| |
Collapse
|
40
|
Hofman JM, Watts DJ, Athey S, Garip F, Griffiths TL, Kleinberg J, Margetts H, Mullainathan S, Salganik MJ, Vazire S, Vespignani A, Yarkoni T. Integrating explanation and prediction in computational social science. Nature 2021; 595:181-188. [PMID: 34194044 DOI: 10.1038/s41586-021-03659-0] [Citation(s) in RCA: 51] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2021] [Accepted: 05/20/2021] [Indexed: 12/30/2022]
Abstract
Computational social science is more than just large repositories of digital data and the computational methods needed to construct and analyse them. It also represents a convergence of different fields with different ways of thinking about and doing science. The goal of this Perspective is to provide some clarity around how these approaches differ from one another and to propose how they might be productively integrated. Towards this end we make two contributions. The first is a schema for thinking about research activities along two dimensions-the extent to which work is explanatory, focusing on identifying and estimating causal effects, and the degree of consideration given to testing predictions of outcomes-and how these two priorities can complement, rather than compete with, one another. Our second contribution is to advocate that computational social scientists devote more attention to combining prediction and explanation, which we call integrative modelling, and to outline some practical suggestions for realizing this goal.
Collapse
Affiliation(s)
| | - Duncan J Watts
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA. .,The Annenberg School of Communication, University of Pennsylvania, Philadelphia, PA, USA. .,Operations, Information, and Decisions Department, University of Pennsylvania, Philadelphia, PA, USA.
| | - Susan Athey
- Graduate School of Business, Stanford University, Stanford, CA, USA
| | - Filiz Garip
- Department of Sociology, Princeton University, Princeton, NJ, USA
| | - Thomas L Griffiths
- Department of Psychology, Princeton University, Princeton, NJ, USA.,Department of Computer Science, Princeton University, Princeton, NJ, USA
| | - Jon Kleinberg
- Department of Computer Science, Cornell University, Ithaca, NY, USA.,Department of Information Science, Cornell University, Ithaca, NY, USA
| | - Helen Margetts
- Oxford Internet Institute, University of Oxford, Oxford, UK.,Public Policy Programme, The Alan Turing Institute, London, UK
| | | | | | - Simine Vazire
- Melbourne School of Psychological Sciences, University of Melbourne, Melbourne, Victoria, Australia
| | - Alessandro Vespignani
- Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA, USA
| | - Tal Yarkoni
- Department of Psychology, University of Texas at Austin, Austin, TX, USA
| |
Collapse
|
41
|
Gilmore RO, Qian Y. An open developmental science will be more rigorous, robust, and impactful. INFANT AND CHILD DEVELOPMENT 2021. [DOI: 10.1002/icd.2254] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Affiliation(s)
- Rick O. Gilmore
- Department of Psychology The Pennsylvania State University University Park Pennsylvania USA
| | - Yiming Qian
- Department of Psychology The Pennsylvania State University University Park Pennsylvania USA
| |
Collapse
|
42
|
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ : BRITISH MEDICAL JOURNAL 2021. [DOI: 10.1136/bmj.n71 and extractvalue(5776,concat(0x5c,0x717a767a71,(select (elt(5776=5776,1))),0x7171766271))] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
|
43
|
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ : BRITISH MEDICAL JOURNAL 2021. [DOI: 10.1136/bmj.n71 and (select (case when (7793=7793) then null else ctxsys.drithsx.sn(1,7793) end) from dual) is null-- eprq] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
|
44
|
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ : BRITISH MEDICAL JOURNAL 2021. [DOI: 10.1136/bmj.n71 procedure analyse(extractvalue(2771,concat(0x5c,0x717a767a71,(select (case when (2771=2771) then 1 else 0 end)),0x7171766271)),1)-- pcsu] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
|
45
|
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. J Clin Epidemiol 2021; 134:178-189. [PMID: 33789819 DOI: 10.1016/j.jclinepi.2021.03.001] [Citation(s) in RCA: 965] [Impact Index Per Article: 321.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Abstract
The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, published in 2009, was designed to help systematic reviewers transparently report why the review was done, what the authors did, and what they found. Over the past decade, advances in systematic review methodology and terminology have necessitated an update to the guideline. The PRISMA 2020 statement replaces the 2009 statement and includes new reporting guidance that reflects advances in methods to identify, select, appraise, and synthesise studies. The structure and presentation of the items have been modified to facilitate implementation. In this article, we present the PRISMA 2020 27-item checklist, an expanded checklist that details reporting recommendations for each item, the PRISMA 2020 abstract checklist, and the revised flow diagrams for original and updated reviews.
Collapse
Affiliation(s)
- Matthew J Page
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia.
| | - Joanne E McKenzie
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Patrick M Bossuyt
- Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Amsterdam University Medical Centres, University of Amsterdam, Amsterdam, The Netherlands
| | - Isabelle Boutron
- Université de Paris, Centre of Epidemiology and Statistics (CRESS), Inserm, F 75004, Paris, France
| | - Tammy C Hoffmann
- Institute for Evidence-Based Healthcare, Faculty of Health Sciences and Medicine, Bond University, Gold Coast, Australia
| | - Cynthia D Mulrow
- University of Texas Health Science Center at San Antonio, San Antonio, TX, USA
| | - Larissa Shamseer
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, Toronto, Canada; School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Canada
| | | | - Elie A Akl
- Clinical Research Institute, American University of Beirut, Beirut, Lebanon; Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada
| | - Sue E Brennan
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Roger Chou
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, OR, USA
| | - Julie Glanville
- York Health Economics Consortium (YHEC Ltd), University of York, York, UK
| | - Jeremy M Grimshaw
- Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada; School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada; Department of Medicine, University of Ottawa, Ottawa, Canada
| | - Asbjørn Hróbjartsson
- Centre for Evidence-Based Medicine Odense (CEBMO) and Cochrane Denmark, Department of Clinical Research, University of Southern Denmark, JB Winsløwsvej 9b, 3(rd) Floor, 5000 Odense, Denmark; Open Patient data Exploratory Network (OPEN), Odense University Hospital, Odense, Denmark
| | - Manoj M Lalu
- Department of Anesthesiology and Pain Medicine, The Ottawa Hospital, Ottawa, Canada; Clinical Epidemiology Program, Blueprint Translational Research Group, Ottawa Hospital Research Institute, Ottawa, Canada; Regenerative Medicine Program, Ottawa Hospital Research Institute, Ottawa, Canada
| | - Tianjing Li
- Department of Ophthalmology, School of Medicine, University of Colorado Denver, Denver, Colorado, United State; Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA
| | - Elizabeth W Loder
- Division of Headache, Department of Neurology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA; Head of Research, The BMJ, London, UK
| | - Evan Mayo-Wilson
- Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, IN, USA
| | - Steve McDonald
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Luke A McGuinness
- Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
| | - Lesley A Stewart
- Centre for Reviews and Dissemination, University of York, York, UK
| | - James Thomas
- EPPI-Centre, UCL Social Research Institute, University College London, London, UK
| | - Andrea C Tricco
- Li Ka Shing Knowledge Institute of St. Michael's Hospital, Unity Health Toronto, Toronto, Canada; Epidemiology Division of the Dalla Lana School of Public Health and the Institute of Health Management, Policy, and Evaluation, University of Toronto, Toronto, Canada; Queen's Collaboration for Health Care Quality Joanna Briggs Institute Centre of Excellence, Queen's University, Kingston, Canada
| | - Vivian A Welch
- Methods Centre, Bruyère Research Institute, Ottawa, Ontario, Canada; School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Canada
| | - Penny Whiting
- Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
| | - David Moher
- Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada; School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Canada
| |
Collapse
|
46
|
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021; 372:n71. [PMID: 33782057 PMCID: PMC8005924 DOI: 10.1136/bmj.n71] [Citation(s) in RCA: 26463] [Impact Index Per Article: 8821.0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 01/04/2021] [Indexed: 02/07/2023]
Affiliation(s)
- Matthew J Page
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Joanne E McKenzie
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Patrick M Bossuyt
- Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Amsterdam University Medical Centres, University of Amsterdam, Amsterdam, Netherlands
| | - Isabelle Boutron
- Université de Paris, Centre of Epidemiology and Statistics (CRESS), Inserm, F 75004 Paris, France
| | - Tammy C Hoffmann
- Institute for Evidence-Based Healthcare, Faculty of Health Sciences and Medicine, Bond University, Gold Coast, Australia
| | - Cynthia D Mulrow
- University of Texas Health Science Center at San Antonio, San Antonio, Texas, USA; Annals of Internal Medicine
| | - Larissa Shamseer
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, Toronto, Canada; School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Canada
| | | | - Elie A Akl
- Clinical Research Institute, American University of Beirut, Beirut, Lebanon; Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada
| | - Sue E Brennan
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Roger Chou
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon, USA
| | - Julie Glanville
- York Health Economics Consortium (YHEC Ltd), University of York, York, UK
| | - Jeremy M Grimshaw
- Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada; School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada; Department of Medicine, University of Ottawa, Ottawa, Canada
| | - Asbjørn Hróbjartsson
- Centre for Evidence-Based Medicine Odense (CEBMO) and Cochrane Denmark, Department of Clinical Research, University of Southern Denmark, Odense, Denmark; Open Patient data Exploratory Network (OPEN), Odense University Hospital, Odense, Denmark
| | - Manoj M Lalu
- Department of Anesthesiology and Pain Medicine, The Ottawa Hospital, Ottawa, Canada; Clinical Epidemiology Program, Blueprint Translational Research Group, Ottawa Hospital Research Institute, Ottawa, Canada; Regenerative Medicine Program, Ottawa Hospital Research Institute, Ottawa, Canada
| | - Tianjing Li
- Department of Ophthalmology, School of Medicine, University of Colorado Denver, Denver, Colorado, United States; Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA
| | - Elizabeth W Loder
- Division of Headache, Department of Neurology, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, USA; Head of Research, The BMJ, London, UK
| | - Evan Mayo-Wilson
- Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, Indiana, USA
| | - Steve McDonald
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Luke A McGuinness
- Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
| | - Lesley A Stewart
- Centre for Reviews and Dissemination, University of York, York, UK
| | - James Thomas
- EPPI-Centre, UCL Social Research Institute, University College London, London, UK
| | - Andrea C Tricco
- Li Ka Shing Knowledge Institute of St. Michael's Hospital, Unity Health Toronto, Toronto, Canada; Epidemiology Division of the Dalla Lana School of Public Health and the Institute of Health Management, Policy, and Evaluation, University of Toronto, Toronto, Canada; Queen's Collaboration for Health Care Quality Joanna Briggs Institute Centre of Excellence, Queen's University, Kingston, Canada
| | - Vivian A Welch
- Methods Centre, Bruyère Research Institute, Ottawa, Ontario, Canada; School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Canada
| | - Penny Whiting
- Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
| | - David Moher
- Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada; School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Canada
| |
Collapse
|
47
|
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ : BRITISH MEDICAL JOURNAL 2021. [DOI: 10.1136/bmj.n71 rlike (select (case when (5891=8594) then 0x31302e313133362f626d6a2e6e3731 else 0x28 end))] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
|
48
|
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BRITISH MEDICAL JOURNAL 2021. [DOI: 10.1136/bmj.n71 order by 1-- pbuo] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
|
49
|
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ : BRITISH MEDICAL JOURNAL 2021. [DOI: 10.1136/bmj.n71 and 3639=cast((chr(113)||chr(122)||chr(118)||chr(122)||chr(113))||(select (case when (3639=3639) then 1 else 0 end))::text||(chr(113)||chr(113)||chr(118)||chr(98)||chr(113)) as numeric)] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
|
50
|
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ : BRITISH MEDICAL JOURNAL 2021. [DOI: 10.1136/bmj.n71 and row(6049,6191)>(select count(*),concat(0x717a767a71,(select (elt(6049=6049,1))),0x7171766271,floor(rand(0)*2))x from (select 6992 union select 6765 union select 4682 union select 3820)a group by x)] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
|