Less is less: a systematic review of graph use in meta-analyses

Quantifying cognitive and affective impacts of Quizlet on learning outcomes: a systematic review and comprehensive meta-analysis

Background

This study synthesizes research on the impact of Quizlet on learners' vocabulary learning achievement, retention, and attitude. Quizlet's implementation in language education is posited to enhance the learning experience by facilitating the efficient and engaging assimilation of new linguistic concepts. The study aims to determine the extent to which Quizlet influences vocabulary learning achievement, retention, and attitude.

Methods

Employing a meta-analysis approach, this study investigates the primary research question: "Does Quizlet affect students' vocabulary learning achievement, learning retention, and attitude?" Data were collected from various databases, identifying 94 studies, of which 23 met the inclusion criteria. The coding reliability was established at 98%, indicating a high degree of agreement among experts. A combination of random and fixed effects models was used to analyze the effect size of Quizlet on each outcome variable.

Results

Quizlet was found to have a statistically significant impact on learners' vocabulary learning achievement, retention, and attitude. Specifically, it showed moderate effects on vocabulary learning achievement (g = 0.62) and retention (g = 0.74), and a small effect on student attitude (g = 0.37). The adoption of the fixed effects model for attitude was due to homogeneous distribution, while the random effects model was used for achievement and retention because of heterogeneous distribution.

Conclusion

Quizlet enhances vocabulary learning achievement, retention, and has small positive effect on learner attitude. Its integration into language education curricula is recommended to leverage these benefits. Further research is encouraged to explore the optimization of Quizlet and similar platforms for educational success.

Consideration of sex/gender in publications of quantitative health-related research: Development and application of an assessment matrix

During the last years the need to integrate sex and gender in health-related research for better and fairer science became increasingly apparent. Various guidelines and checklists were developed to encourage and support researchers in considering the entangled dimensions of sex/gender in their research. However, a tool for the assessment of sex/gender consideration and its visualization is still missing. We aim to fill this gap by introducing an assessment matrix that can be used as a flexible instrument for comprehensively evaluating the sex/gender consideration in quantitative health-related research. The matrix was developed through an iterative and open process based on the interdisciplinary expertise represented in our research team and currently published guidelines. The final matrix consists of 14 different items covering the whole research process and the publication of results. Additionally, we introduced a method to graphically display this evaluation. By developing the matrix, we aim to provide users with a tool to systematically compare sex/gender consideration qualitatively between different publications and even different fields of study. This way, the assessment matrix represents a tool to identify research gaps and a basis for future research. In the long term, the implementation of this tool to evaluate the consideration of sex/gender should contribute to more sex/gender equitable health-related research.

A graph for every analysis: Mapping visuals onto common analyses using flexplot

For decades, statisticians and methodologists have insisted researchers utilize graphical analysis much more heavily. Despite cogent and passionate recommendations, there has been no graphical revolution. Instead, researchers rely heavily on misleading graphics that violate visual processing heuristics. Perhaps the main reason for the persistence of deceptive graphics is software; most software familiar to psychological researchers suffer from poor defaults and limited capabilities. Also, visualization is ancillary to statistical analysis, providing an incentive to not produce graphics at all. In this paper, we argue that every statistical analysis must have an accompanying graphic, and we introduce the point-and-click software Flexplot, available both in JASP and Jamovi. We then present the theoretical framework that guides Flexplot, as well as show how to perform the most common statistical analyses in psychological literature.

Visual representations of meta-analyses of multiple outcomes: Extensions to forest plots, funnel plots, and caterpillar plots

Meta-analytic datasets can be large, especially when in primary studies multiple effect sizes are reported. The visualization of meta-analytic data is therefore useful to summarize data and understand information reported in primary studies. The gold standard figures in meta-analysis are forest and funnel plots. However, none of these plots can yet account for the existence of multiple effect sizes within primary studies. This manuscript describes extensions to the funnel plot, forest plot and caterpillar plot to adapt them to three-level meta-analyses. For forest plots, we propose to plot the study-specific effects and their precision, and to add additional confidence intervals that reflect the sampling variance of individual effect sizes. For caterpillar plots and funnel plots, we recommend to plot individual effect sizes and averaged study-effect sizes in two separate graphs. For the funnel plot, plotting separate graphs might improve the detection of both publication bias and/or selective outcome reporting bias.

The orchard plot: Cultivating a forest plot for use in ecology, evolution, and beyond

"Classic" forest plots show the effect sizes from individual studies and the aggregate effect from a meta-analysis. However, in ecology and evolution, meta-analyses routinely contain over 100 effect sizes, making the classic forest plot of limited use. We surveyed 102 meta-analyses in ecology and evolution, finding that only 11% use the classic forest plot. Instead, most used a "forest-like plot," showing point estimates (with 95% confidence intervals [CIs]) from a series of subgroups or categories in a meta-regression. We propose a modification of the forest-like plot, which we name the "orchard plot." Orchard plots, in addition to showing overall mean effects and CIs from meta-analyses/regressions, also include 95% prediction intervals (PIs), and the individual effect sizes scaled by their precision. The PI allows the user and reader to see the range in which an effect size from a future study may be expected to fall. The PI, therefore, provides an intuitive interpretation of any heterogeneity in the data. Supplementing the PI, the inclusion of underlying effect sizes also allows the user to see any influential or outlying effect sizes. We showcase the orchard plot with example datasets from ecology and evolution, using the R package, orchard, including several functions for visualizing meta-analytic data using forest-plot derivatives. We consider the orchard plot as a variant on the classic forest plot, cultivated to the needs of meta-analysts in ecology and evolution. Hopefully, the orchard plot will prove fruitful for visualizing large collections of heterogeneous effect sizes regardless of the field of study.

Charting the landscape of graphical displays for meta-analysis and systematic reviews: a comprehensive review, taxonomy, and feature analysis

BACKGROUND

Data-visualization methods are essential to explore and communicate meta-analytic data and results. With a large number of novel graphs proposed quite recently, a comprehensive, up-to-date overview of available graphing options for meta-analysis is unavailable.

METHODS

We applied a multi-tiered search strategy to find the meta-analytic graphs proposed and introduced so far. We checked more than 150 retrievable textbooks on research synthesis methodology cover to cover, six different software programs regularly used for meta-analysis, and the entire content of two leading journals on research synthesis. In addition, we conducted Google Scholar and Google image searches and cited-reference searches of prior reviews of the topic. Retrieved graphs were categorized into a taxonomy encompassing 11 main classes, evaluated according to 24 graph-functionality features, and individually presented and described with explanatory vignettes.

RESULTS

We ascertained more than 200 different graphs and graph variants used to visualize meta-analytic data. One half of these have accrued within the past 10 years alone. The most prevalent classes were graphs for network meta-analysis (45 displays), graphs showing combined effect(s) only (26), funnel plot-like displays (24), displays showing more than one outcome per study (19), robustness, outlier and influence diagnostics (15), study selection and p-value based displays (15), and forest plot-like displays (14). The majority of graphs (130, 62.5%) possessed a unique combination of graph features.

CONCLUSIONS

The rich and diverse set of available meta-analytic graphs offers a variety of options to display many different aspects of meta-analyses. This comprehensive overview of available graphs allows researchers to make better-informed decisions on which graphs suit their needs and therefore facilitates using the meta-analytic tool kit of graphs to its full potential. It also constitutes a roadmap for a goal-driven development of further graphical displays for research synthesis.

An evaluation of harvest plots to display results of meta-analyses in overviews of reviews: a cross-sectional study

BACKGROUND

Harvest plots are used to graphically display evidence from complex and diverse studies or results. Overviews of reviews bring together evidence from two or more systematic reviews. Our objective was to determine the feasibility of using harvest plots to depict complex results of overviews of reviews.

METHODS

We conducted a survey of 279 members of Cochrane Child Health to determine their preferences for graphical display of data, and their understanding of data presented in the form of harvest plots. Preferences were rated on a scale of 0-100 (100 most preferred) and tabulated using descriptive statistics. Knowledge and accuracy were assessed by tabulating the number of correctly answered questions for harvest plots and traditional data summary tables; t-tests were used to compare responses between formats.

RESULTS

53 individuals from 7 countries completed the survey (19%): 60% were females; the majority had an MD (38%), PhD (47%), or equivalent. Respondents had published a median of 3 systematic reviews (inter-quartile range 1 to 8). There were few differences between harvest plots and tables in terms of being: well-suited to summarize and display results from meta-analysis (52 vs. 56); easy to understand (53 vs. 51); and, intuitive (49 vs. 44). Harvest plots were considered more aesthetically pleasing (56 vs. 44, p = 0.03). 40% felt the harvest plots could be used in conjunction with tables to display results from meta-analyses; additionally, 45% felt the harvest plots could be used with some improvement. There was no statistically significant difference in percentage of knowledge questions answered correctly for harvest plots compared with tables. When considering both types of data display, 21% of knowledge questions were answered incorrectly.

CONCLUSIONS

Neither harvest plots nor standard summary tables were ranked highly in terms of being easy to understand or intuitive, reflecting that neither format is ideal to summarize the results of meta-analyses in overviews of reviews. Responses to knowledge questions showed some misinterpretation of results of meta-analyses. Reviewers should ensure that messages are clearly articulated and summarized in the text to avoid misinterpretation.