1
|
Kaufmann E, Reips UD. Meta-analysis in a digitalized world: A step-by-step primer. Behav Res Methods 2024; 56:1-21. [PMID: 38575774 PMCID: PMC11362208 DOI: 10.3758/s13428-024-02374-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/17/2024] [Indexed: 04/06/2024]
Abstract
In recent years, much research and many data sources have become digital. Some advantages of digital or Internet-based research, compared to traditional lab research (e.g., comprehensive data collection and storage, availability of data) are ideal for an improved meta-analyses approach.In the meantime, in meta-analyses research, different types of meta-analyses have been developed to provide research syntheses with accurate quantitative estimations. Due to its rich and unique palette of corrections, we recommend to using the Schmidt and Hunter approach for meta-analyses in a digitalized world. Our primer shows in a step-by-step fashion how to conduct a high quality meta-analysis considering digital data and highlights the most obvious pitfalls (e.g., using only a bare-bones meta-analysis, no data comparison) not only in aggregation of the data, but also in the literature search and coding procedure which are essential steps in any meta-analysis. Thus, this primer of meta-analyses is especially suited for a situation where much of future research is headed to: digital research. To map Internet-based research and to reveal any research gap, we further synthesize meta-analyses on Internet-based research (15 articles containing 24 different meta-analyses, on 745 studies, with 1,601 effect sizes), resulting in the first mega meta-analysis of the field. We found a lack of individual participant data (e.g., age and nationality). Hence, we provide a primer for high-quality meta-analyses and mega meta-analyses that applies to much of coming research and also basic hands-on knowledge to conduct or judge the quality of a meta-analyses in a digitalized world.
Collapse
Affiliation(s)
- Esther Kaufmann
- Research Methods, Assessment, and iScience, Department of Psychology, University of Konstanz, Konstanz, Germany.
| | - Ulf-Dietrich Reips
- Research Methods, Assessment, and iScience, Department of Psychology, University of Konstanz, Konstanz, Germany
| |
Collapse
|
2
|
Kaufmann E. Teachers' judgment accuracy: A replication check by psychometric meta-analysis. PLoS One 2024; 19:e0307594. [PMID: 39052673 PMCID: PMC11271880 DOI: 10.1371/journal.pone.0307594] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/02/2024] [Accepted: 07/08/2024] [Indexed: 07/27/2024] Open
Abstract
Teachers' judgment accuracy is a core competency in their daily business. Due to its importance, several meta-analyses have estimated how accurately teachers judge students' academic achievements by measuring teachers' judgment accuracy (i.e., the correlation between teachers' judgments of students' academic abilities and students' scores on achievement tests). In our study, we considered previous meta-analyses and updated these databases and the analytic combination of data using a psychometric meta-analysis to explain variations in results across studies. Our results demonstrate the importance of considering aggregation and publication bias as well as correcting for the most important artifacts (e.g., sampling and measurement error), but also that most studies fail to report the data needed for conducting a meta-analysis according to current best practices. We find that previous reviews have underestimated teachers' judgment accuracy and overestimated the variance in estimates of teachers' judgment accuracy across studies because at least 10% of this variance may be associated with common artifacts. We conclude that ignoring artifacts, as in classical meta-analysis, may lead one to erroneously conclude that moderator variables, instead of artifacts, explain any variation. We describe how online data repositories could improve the scientific process and the potential for using psychometric meta-analysis to synthesize results and assess replicability.
Collapse
Affiliation(s)
- Esther Kaufmann
- Research Methods, Assessment and iScience, Department of Psychology, University of Konstanz, Konstanz, Germany
| |
Collapse
|
3
|
Advanced methods and implementations for the meta-analyses of animal models: Current practices and future recommendations. Neurosci Biobehav Rev 2023; 146:105016. [PMID: 36566804 DOI: 10.1016/j.neubiorev.2022.105016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2022] [Revised: 12/19/2022] [Accepted: 12/20/2022] [Indexed: 12/24/2022]
Abstract
Meta-analytic techniques have been widely used to synthesize data from animal models of human diseases and conditions, but these analyses often face two statistical challenges due to complex nature of animal data (e.g., multiple effect sizes and multiple species): statistical dependency and confounding heterogeneity. These challenges can lead to unreliable and less informative evidence, which hinders the translation of findings from animal to human studies. We present a literature survey of meta-analysis using animal models (animal meta-analysis), showing that these issues are not adequately addressed in current practice. To address these challenges, we propose a meta-analytic framework based on multilevel (linear mixed-effects) models. Through conceptualization, formulations, and worked examples, we illustrate how this framework can appropriately address these issues while allowing for testing new questions. Additionally, we introduce other advanced techniques such as multivariate models, robust variance estimation, and meta-analysis of emergent effect sizes, which can deliver robust inferences and novel biological insights. We also provide a tutorial with annotated R code to demonstrate the implementation of these techniques.
Collapse
|
4
|
Wolff F, Möller J. An Individual Participant Data Meta-analysis of the Joint Effects of Social, Dimensional, and Temporal Comparisons on Students’ Academic Self-Concepts. EDUCATIONAL PSYCHOLOGY REVIEW 2022. [DOI: 10.1007/s10648-022-09686-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/03/2022]
Abstract
Abstract
Previous research has shown that three comparison types are involved in the formation of students’ academic self-concepts: social comparisons (where students compare their achievement with their classmates), dimensional comparisons (where students compare their achievement in different subjects), and temporal comparisons (where students compare their achievement across time). The 2I/E model provides a framework to describe the joint effects of these comparisons. To date, it has been tested in 12 empirical studies. However, integration of these findings is lacking, especially in terms of yielding reliable estimates of the strength of social, dimensional, and temporal comparison effects. We therefore conducted an individual participant data (IPD) meta-analysis, in which we reanalyzed the data used in all prior 2I/E model studies (N = 45,248). This IPD meta-analysis provided strong support for the 2I/E model: There were moderate social comparison effects, small to moderate dimensional comparison effects, and small temporal comparison effects on students’ math and verbal self-concepts. Moreover, several moderating variables affected the strength of these effects. In particular, the social and temporal comparison effects were stronger in studies using grades instead of test scores as achievement indicators. Older students showed weaker social comparison effects but stronger dimensional comparison effects compared to younger students. Social comparison effects were also stronger in academic track schools compared to nonacademic track schools. Gender and migration background had only very small impacts on the strength of single comparison effects. In sum, this IPD meta-analysis significantly enhances our knowledge of comparison making in the process of students’ self-concept formation.
Collapse
|
5
|
Frech ML, Friese M, Loschelder DD. "My Goal Is to Lose 2.923 kg!"-Efficacy of Precise Versus Round Goals for Body Weight Reduction. Front Psychol 2022; 13:793962. [PMID: 35197900 PMCID: PMC8860075 DOI: 10.3389/fpsyg.2022.793962] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2021] [Accepted: 01/04/2022] [Indexed: 12/18/2022] Open
Abstract
Overweight individuals often struggle to lose weight. While previous studies established goal setting as an effective strategy for weight loss, little is known about the effects of numeric goal precision. The present research investigated whether and how the precision of weight loss goals—the number of trailing zeros—impacts a goal’s effectiveness. In two preregistered, longitudinal experiments, we contrasted competing predictions as to whether precise (e.g., 2.923 kg) or round (e.g., 3.000 kg) goals are more effective compared to a waiting control condition. In Experiment 1 (N = 121), participants in the two goal conditions lost more weight compared to the control condition—an effect that was mainly driven by precise (rather than round) goals. In Experiment 2 (N = 150), we sought to replicate this effect but found no significant weight loss differences. An individual participant data (IPD) meta-analysis across both experiments revealed that (a) the goal groups jointly lost more weight than the waiting control group and (b) the precise and round goal groups did not differ in weight loss success. An IPD-based multiple mediation analysis showed that healthier eating, but not physical exercise accounted for goal-setting-induced weight loss. We discuss possible explanations for the null findings in Experiment 2 and highlight directions for future research.
Collapse
Affiliation(s)
- Marie-Lena Frech
- Institute of Management & Organization, Leuphana University, Lueneburg, Germany
| | - Malte Friese
- Department of Psychology, Saarland University, Saarbruecken, Germany
| | - David D Loschelder
- Institute of Management & Organization, Leuphana University, Lueneburg, Germany
| |
Collapse
|
6
|
Klose M, Steger D, Fick J, Artelt C. Decrypting Log Data. ZEITSCHRIFT FUR PSYCHOLOGIE-JOURNAL OF PSYCHOLOGY 2022. [DOI: 10.1027/2151-2604/a000484] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
Abstract
Abstract. Analyzing log data from digital learning environments provides information about online learning. However, it remains unclear how this information can be transferred to psychologically meaningful variables or how it is linked to learning outcomes. The present study summarizes findings on correlations between general online activity and learning outcomes in university settings. The course format, instructions to engage in online discussions, requirements, operationalization of general online activity, and publication year are considered moderators. A multi-source search provided 41 studies ( N = 28,986) reporting 69 independent samples and 104 effect sizes. The three-level random-effects meta-analysis identified a pooled effect of r = .25 p = .003, 95% CI [.09, .41], indicating that students who are more active online have better grades. Despite high heterogeneity, Q(103) = 3,960.04, p < .001, moderator analyses showed no statistically significant effect. We discuss further potential influencing factors in online courses and highlight the potential of learning analytics.
Collapse
Affiliation(s)
- Maria Klose
- Leibniz Institute for Educational Trajectories (LIfBi), University of Bamberg, Germany
| | - Diana Steger
- Department of Psychological Assessment, University of Kassel, Germany
| | - Julian Fick
- Institute for Communication Science and Institute of Educational Psychology, Technische Universität Braunschweig, Germany
| | - Cordula Artelt
- Leibniz Institute for Educational Trajectories (LIfBi), University of Bamberg, Germany
- Department of Longitudinal Educational Research, University of Bamberg, Germany
| |
Collapse
|
7
|
How accurately do teachers’ judge students? Re-analysis of Hoge and Coladarci (1989) meta-analysis. CONTEMPORARY EDUCATIONAL PSYCHOLOGY 2020. [DOI: 10.1016/j.cedpsych.2020.101902] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
|
8
|
Agyeman AA, Chin KL, Landersdorfer CB, Liew D, Ofori-Asenso R. Smell and Taste Dysfunction in Patients With COVID-19: A Systematic Review and Meta-analysis. Mayo Clin Proc 2020; 95:1621-1631. [PMID: 32753137 PMCID: PMC7275152 DOI: 10.1016/j.mayocp.2020.05.030] [Citation(s) in RCA: 271] [Impact Index Per Article: 67.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/05/2020] [Revised: 05/19/2020] [Accepted: 05/27/2020] [Indexed: 01/08/2023]
Abstract
OBJECTIVE To estimate the prevalence of olfactory and gustatory dysfunctions (OGDs) among patients infected with novel coronavirus disease 2019 (COVID-19). METHODS A systematic review was conducted by searching MEDLINE, EMBASE, and the preprint server MedRxiv from their inception until May 11, 2020, using the terms anosmia or hyposmia or dysosmia or olfactory dysfunction or olfaction disorder or smell dysfunction or ageusia or hypogeusia or dysgeusia or taste dysfunction or gustatory dysfunction or neurological and COVID-19 or 2019 novel coronavirus or 2019-nCoV or SARS-CoV-2. The references of included studies were also manually screened. Only studies involving patients with diagnostic-confirmed COVID-19 infection were included. Random-effects meta-analysis was performed. RESULTS Twenty-four studies with data from 8438 patients with test-confirmed COVID-19 infection from 13 countries were included. The pooled proportions of patients presenting with olfactory dysfunction and gustatory dysfunction were 41.0% (95% CI, 28.5% to 53.9%) and 38.2% (95% CI, 24.0% to 53.6%), respectively. Increasing mean age correlated with lower prevalence of olfactory (coefficient = -0.076; P=.02) and gustatory (coefficient = -0.073; P=.03) dysfunctions. There was a higher prevalence of olfactory dysfunctions with the use of objective measurements compared with self-reports (coefficient = 2.33; P=.01). No significant moderation of the prevalence of OGDs by sex was observed. CONCLUSION There is a high prevalence of OGDs among patients infected with COVID-19. Routine screening for these conditions could contribute to improved case detection in the ongoing COVID-19 pandemic. However, to better inform population screening measures, further studies are needed to establish causality.
Collapse
Affiliation(s)
- Akosua Adom Agyeman
- Centre for Medicine Use and Safety, Monash Institute of Pharmaceutical Sciences, Melbourne, Victoria, Australia
| | - Ken L Chin
- Department of Epidemiology and Preventive Medicine, Melbourne, Victoria, Australia; Monash University, Melbourne Medical School, University of Melbourne, Melbourne, Victoria, Australia
| | - Cornelia B Landersdorfer
- Centre for Medicine Use and Safety, Monash Institute of Pharmaceutical Sciences, Melbourne, Victoria, Australia
| | - Danny Liew
- Department of Epidemiology and Preventive Medicine, Melbourne, Victoria, Australia
| | - Richard Ofori-Asenso
- Department of Epidemiology and Preventive Medicine, Melbourne, Victoria, Australia; Department of Pharmacy, Faculty of Health and Medical Science, University of Copenhagen, Copenhagen, Denmark.
| |
Collapse
|
9
|
Abstract
Much of the carer literature has focused on depression and burden as primary outcomes and anxiety appear somewhat neglected. Providing evidence on the prevalence of carer anxiety is critical as it can enhance awareness among professionals, which in turn can lead to improved access to efficacious treatments. This meta-analysis updated the previous review conducted in 2007 to estimate the up-to-date prevalence of anxiety in informal carers for people with dementia. Literature searches were conducted in databases of published and unpublished literature. Events and sample size data were pooled using a random effects model to obtain an overall prevalence percentage. A total of 10 studies were included, resulting in a pooled estimate of anxiety prevalence at 32.1% (95% confidence interval: 20.6%-46.2%, P = .01). Significant heterogeneity was found, which was not reduced following sensitivity analysis. This study suggests anxiety is a prevalent difficulty experienced by dementia carers. Additional research recommendations and clinical implications are discussed.
Collapse
Affiliation(s)
- Laura Kaddour
- Department of Clinical Psychology, Norwich Medical School, University of East Anglia, Norwich, United Kingdom
| | - Naoko Kishita
- School of Health Sciences, University of East Anglia, Norwich, United Kingdom
| |
Collapse
|
10
|
Steger D, Schroeders U, Gnambs T. A Meta-Analysis of Test Scores in Proctored and Unproctored Ability Assessments. EUROPEAN JOURNAL OF PSYCHOLOGICAL ASSESSMENT 2020. [DOI: 10.1027/1015-5759/a000494] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
Abstract
Abstract. Unproctored, web-based assessments are frequently compromised by a lack of control over the participants’ test-taking behavior. It is likely that participants cheat if personal consequences are high. This meta-analysis summarizes findings on context effects in unproctored and proctored ability assessments and examines mean score differences and correlations between both assessment contexts. As potential moderators, we consider (a) the perceived consequences of the assessment, (b) countermeasures against cheating, (c) the susceptibility to cheating of the measure itself, and (d) the use of different test media. For standardized mean differences, a three-level random-effects meta-analysis based on 109 effect sizes from 49 studies (total N = 100,434) identified a pooled effect of Δ = 0.20, 95% CI [0.10, 0.31], indicating higher scores in unproctored assessments. Moderator analyses revealed significantly smaller effects for measures that are difficult to research on the Internet. These results demonstrate that unproctored ability assessments are biased by cheating. Unproctored assessments may be most suitable for tasks that are difficult to search on the Internet.
Collapse
Affiliation(s)
- Diana Steger
- Bamberg Graduate School of Social Sciences, University of Bamberg, Germany
- Department for Individual Differences and Psychological Assessment, Ulm University, Germany
| | - Ulrich Schroeders
- Department of Psychological Assessment, University of Kassel, Germany
| | - Timo Gnambs
- Educational Measurement, Leibniz Institute for Educational Trajectories, Germany
- Institute for Education and Psychology, Johannes Kepler University Linz, Austria
| |
Collapse
|
11
|
Bošnjak M, Erdfelder E. Hotspots in Psychology – 2018 Edition. ZEITSCHRIFT FUR PSYCHOLOGIE-JOURNAL OF PSYCHOLOGY 2018. [DOI: 10.1027/2151-2604/a000323] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
Affiliation(s)
- Michael Bošnjak
- Leibniz Institute for Psychology Information (ZPID), Trier, Germany
| | - Edgar Erdfelder
- University of Mannheim, Department of Psychology, Mannheim, Germany
| |
Collapse
|
12
|
Gnambs T, Scharl A, Schroeders U. The Structure of the Rosenberg Self-Esteem Scale. ZEITSCHRIFT FUR PSYCHOLOGIE-JOURNAL OF PSYCHOLOGY 2018. [DOI: 10.1027/2151-2604/a000317] [Citation(s) in RCA: 26] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
Abstract
Abstract. The Rosenberg Self-Esteem Scale (RSES; Rosenberg, 1965 ) intends to measure a single dominant factor representing global self-esteem. However, several studies have identified some form of multidimensionality for the RSES. Therefore, we examined the factor structure of the RSES with a fixed-effects meta-analytic structural equation modeling approach including 113 independent samples (N = 140,671). A confirmatory bifactor model with specific factors for positively and negatively worded items and a general self-esteem factor fitted best. However, the general factor captured most of the explained common variance in the RSES, whereas the specific factors accounted for less than 15%. The general factor loadings were invariant across samples from the United States and other highly individualistic countries, but lower for less individualistic countries. Thus, although the RSES essentially represents a unidimensional scale, cross-cultural comparisons might not be justified because the cultural background of the respondents affects the interpretation of the items.
Collapse
Affiliation(s)
- Timo Gnambs
- Leibniz Institute for Educational Trajectories, Bamberg, Germany
| | - Anna Scharl
- Leibniz Institute for Educational Trajectories, Bamberg, Germany
| | | |
Collapse
|
13
|
Affiliation(s)
- Edgar Erdfelder
- Department of Psychology, School of Social Sciences, University of Mannheim, Germany
| | - Michael Bošnjak
- Department of Psychology, School of Social Sciences, University of Mannheim, Germany
| |
Collapse
|