Neglect of publication bias compromises meta-analyses of educational research

What works in radiology education for medical students: a systematic review and meta-analysis

BACKGROUND

Medical imaging related knowledge and skills are widely used in clinical practice. However, radiology teaching methods and resultant knowledge among medical students and junior doctors is variable. A systematic review and meta-analysis was performed to compare the impact of different components of radiology teaching methods (active versus passive teaching, eLearning versus traditional face-to-face teaching) on radiology knowledge / skills of medical students.

METHODS

PubMed and Scopus databases were searched for articles published in English over a 15-year period ending in June 2021 quantitatively comparing the effectiveness of undergraduate medical radiology education programs regarding acquisition of knowledge and/or skills. Study quality was appraised by the Medical Education Research Study Quality Instrument (MERSQI) scoring and analyses performed to assess for risk of bias. A random effects meta-analysis was performed to pool weighted effect sizes across studies and I2 statistics quantified heterogeneity. A meta-regression analysis was performed to assess for sources of heterogeneity.

RESULTS

From 3,052 articles, 40 articles involving 6,242 medical students met inclusion criteria. Median MERSQI score of the included articles was 13 out of 18 possible with moderate degree of heterogeneity (I2 = 93.42%). Thematic analysis suggests trends toward synergisms between radiology and anatomy teaching, active learning producing superior knowledge gains compared with passive learning and eLearning producing equivalent learning gains to face-to-face teaching. No significant differences were detected in the effectiveness of methods of radiology education. However, when considered with the thematic analysis, eLearning is at least equivalent to traditional face-to-face teaching and could be synergistic.

CONCLUSIONS

Studies of educational interventions are inherently heterogeneous and contextual, typically tailored to specific groups of students. Thus, we could not draw definitive conclusion about effectiveness of the various radiology education interventions based on the currently available data. Better standardisation in the design and implementation of radiology educational interventions and design of radiology education research are needed to understand aspects of educational design and delivery that are optimal for learning.

TRIAL REGISTRATION

Prospero registration number CRD42022298607.

Risk and protective factors for (internet) gaming disorder: A meta-analysis of pre-COVID studies

This large-scale meta-analysis aimed to provide the most comprehensive synthesis to date of the available evidence from the pre-COVID period on risk and protective factors for (internet) gaming disorder (as defined in the DSM-5 or ICD-11) across all studied populations. The risk/protective factors included demographic characteristics, psychological, psychopathological, social, and gaming-related factors. In total, we have included 1,586 effects from 253 different studies, summarizing data from 210,557 participants. Apart from estimating these predictive associations and relevant moderating effects, we implemented state-of-the-art adjustments for publication bias, psychometric artifacts, and other forms of bias arising from the publication process. Additionally, we carried out an in-depth assessment of the quality of underlying evidence by examining indications of selective reporting, statistical inconsistencies, the typical power of utilized study designs to detect theoretically relevant effects, and performed various sensitivity analyses. The available evidence suggests the existence of numerous moderately strong and highly heterogeneous risk factors (e.g., male gender, depression, impulsivity, anxiety, stress, gaming time, escape motivation, or excessive use of social networks) but only a few empirically robust protective factors (self-esteem, intelligence, life satisfaction, and education; all having markedly smaller effect sizes). We discuss the theoretical implications of our results for prominent theoretical models of gaming disorder and for the existing and future prevention strategies. The impact of various examined biasing factors on the available evidence seemed to be modest, yet we identified shortcomings in the measurement and reporting practices.

The efficacy of self-guided internet and mobile-based interventions for preventing anxiety and depression - A systematic review and meta-analysis

BACKGROUND

Anxiety and depression are highly prevalent mental disorders which are associated with a considerable personal and economic burden. As treatment alone has a minimal impact on prevalence, there is now a growing focus on interventions which may help prevent anxiety and depression. Internet and mobile based interventions have been identified as a useful avenue for the delivery of preventative programmes due to their scalability and accessibility. The efficacy of interventions that do not require additional support from a trained professional (self-guided) in this capacity is yet to be explored.

METHOD

A systematic search was conducted on the Cochrane Library, PubMed, PsycARTICLES, PsycINFO, OVID, MEDline, PsycEXTRA and SCOPUS databases. Studies were selected according to defined inclusion and exclusion criteria. The primary outcome was evaluating the effect of self-guided internet and mobile based interventions on incidence of anxiety and depression. The secondary outcome was effect on symptom severity.

RESULTS

After identifying and removing duplicates, 3211 studies were screened, 32 of which were eligible for inclusion in the final analysis. Nine studies also reported incidence data (depression = 7, anxiety = 2). The overall Risk Ratios for incidence of anxiety and depression were 0.86 (95% CI [0.28, 2.66], p = .79) and 0.67 (95% CI [0.48, 0.93], p = .02) respectively. Analysis for 27 studies reporting severity of depressive symptoms revealed a significant posttreatment standardised mean difference of -0.27 (95% CI [ -0.37, -0.17], p < .001) for self-guided intervention groups relative to controls. A similar result was observed for 29 studies reporting severity of anxiety symptoms with a standardised mean difference of -0.21 (95% CI [-0.31, -0.10], p < .001).

CONCLUSIONS

Self-guided internet and mobile based interventions appear to be effective at preventing incidence of depression, though further examination of the data suggests that generalisability of this finding may be limited. While self-guided interventions also appear effective in reducing symptoms of anxiety and depression, their ability to prevent incidence of anxiety is less clear. A heavy reliance on symptom measures in the data analysed suggests future research could benefit from prioritising the use of standardised diagnostic measuring tools to assess incidence. Future systematic reviews should aim to include more data from grey literature and reduce the impact of study heterogeneity.

An artificial neural network (ANN) model for publication bias: a machine learning-based study on PubMed meta-analyses

Purpose

No study has investigated the effects of different parameters on publication bias in meta-analyses using a machine learning approach. Therefore, this study aims to evaluate the impact of various factors on publication bias in meta-analyses.

Design/methodology/approach

An electronic questionnaire was created according to some factors extracted from the Cochrane Handbook and AMSTAR-2 tool to identify factors affecting publication bias. Twelve experts were consulted to determine their opinion on the importance of each factor. Each component was evaluated based on its content validity ratio (CVR). In total, 616 meta-analyses comprising 1893 outcomes from PubMed that assessed the presence of publication bias in their reported outcomes were randomly selected to extract their data. The multilayer perceptron (MLP) technique was used in IBM SPSS Modeler 18.0 to construct a prediction model. 70, 15 and 15% of the data were used for the model's training, testing and validation partitions.

Findings

There was a publication bias in 968 (51.14%) outcomes. The established model had an accuracy rate of 86.1%, and all pre-selected nine variables were included in the model. The results showed that the number of databases searched was the most important predictive variable (0.26), followed by the number of searches in the grey literature (0.24), search in Medline (0.17) and advanced search with numerous operators (0.13).

Practical implications

The results of this study can help clinical researchers minimize publication bias in their studies, leading to improved evidence-based medicine.

Originality/value

To the best of the author’s knowledge, this is the first study to model publication bias using machine learning.

Quality and constructed knowledge: Truth, paradigms, and the state of the science

CONTEXT AND TRUTH

Education is a social science. Social science knowledge is related to its context of origin. The concept of global 'truth' in education is therefore of limited use when truth is tempered by context. The wider applicability of our knowledge can only be judged if we look at the context in which that knowledge was produced and the assumptions that underpin it. This calls into question the idea that educational research is a quest for global 'truth', although in relation to programme evaluation, truth tied to context is an aim. An analysis is presented of the effects of social construction on research and evaluation processes, on the selection of paradigms, reporting and interpreting findings, and on the ethics of all this.

QUALITY AND IMPROVEMENT

Quality improvement is based on information selected, constructed and interpreted by those who gather, analyse or use it. The strength, and not the weakness, of our knowledge is that it is socially constructed, contextual and of its time. Increasingly looking for our own truth about educational quality, and not importing the truth of others, is crucial to the state of the science. In terms of quality development, using others' findings must be based on informed local judgement. In social science, those judgements are linked to social context and their associated ideologies.

IMPLICATIONS FOR FUTURE WORK

The hallmark of social science is not a narrowing of focus and the search for one truth, but is a broadening of concepts, theories, paradigms, reported experience and method, and an intention for each to tell their own truth well. This will lead to a wealth of diverse views and analysed experience. The science of medical education must seek many truths.

Publication bias in simulation model studies: The case of ethanol literature

In this study, we explore the potential for publication bias using market simulation results that estimate the effect of US ethanol expansion on corn prices. We provide a new test of whether the publication process routes market simulation results into one of the following two narratives: food-versus-fuel or greenhouse gas (GHG) emissions. Our research question is whether model results with either high price or large land impact are favored for publication in one body of literature or the other. In other words, a model that generates larger price effects might be more readily published in the food-versus-fuel literature while a model that generates larger land use change and GHG emissions might find a home in the GHG emission literature. We develop a test for publication bias based on matching narrative and normalized price effects from simulated market models. As such, our approach differs from past studies of publication bias that typically focus on statistically estimated parameters. This focus could have broad implications: if in the future more studies assess publication bias of quantitative results that are not statistically estimated parameters, then important inferences about publication bias could be drawn. More specifically, such a body of literature could explore the potential that practices common in either statistical methods or other methods tend to encourage or deter publication bias. Turning back to the present case, our findings in this study do not detect a relationship between food-versus-fuel or GHG narrative orientation and corn price effects. The results are relevant to debates about biofuel impacts and our approach can inform the publication bias literature more generally.

Residual Disease After Primary Surgical Treatment for Advanced Epithelial Ovarian Cancer, Part 2: Network Meta-analysis Incorporating Expert Elicitation to Adjust for Publication Bias

BACKGROUND

Previous work has identified a strong association between the achievements of macroscopic cytoreduction and improved overall survival (OS) after primary surgical treatment of advanced epithelial ovarian cancer. Despite the use of contemporary methodology, resulting in the most comprehensive currently available evidence to date in this area, opponents remain skeptical.

AREAS OF UNCERTAINTY

We aimed to conduct sensitivity analyses to adjust for potential publication bias, to confirm or refute existing conclusions and recommendations, leveraging elicitation to incorporate expert opinion. We recommend our approach as an exemplar that should be adopted in other areas of research.

DATA SOURCES

We conducted random-effects network meta-analyses in frequentist and Bayesian (using Markov Chain Montel Carlo simulation) frameworks comparing OS across residual disease thresholds in women with advanced epithelial ovarian cancer after primary cytoreductive surgery. Elicitation methods among experts in gynecology were used to derive priors for an extension to a previously reported Copas selection model and a novel approach using effect estimates calculated from the elicitation exercise, to attempt to adjust for publication bias and increase confidence in the certainty of the evidence.

THERAPEUTIC ADVANCES

Analyses using data from 25 studies (n = 20,927 women) all showed the prognostic importance of complete cytoreduction (0 cm) in both frameworks. Experts accepted publication bias was likely, but after adjustment for their opinions, published results overpowered the informative priors incorporated into the Bayesian sensitivity analyses. Effect estimates were attenuated but conclusions were robust in all analyses.

CONCLUSIONS

There remains a strong association between the achievement of complete cytoreduction and improved OS even after adjustment for publication bias using strong informative priors formed from an expert elicitation exercise. The concepts of the elicitation survey should be strongly considered for utilization in other meta-analyses.

Using selection models to assess sensitivity to publication bias: A tutorial and call for more routine use

In meta-analyses, it is critical to assess the extent to which publication bias might have compromised the results. Classical methods based on the funnel plot, including Egger's test and Trim-and-Fill, have become the de facto default methods to do so, with a large majority of recent meta-analyses in top medical journals (85%) assessing for publication bias exclusively using these methods. However, these classical funnel plot methods have important limitations when used as the sole means of assessing publication bias: they essentially assume that the publication process favors large point estimates for small studies and does not affect the largest studies, and they can perform poorly when effects are heterogeneous. In light of these limitations, we recommend that meta-analyses routinely apply other publication bias methods in addition to or instead of classical funnel plot methods. To this end, we describe how to use and interpret selection models. These methods make the often more realistic assumption that publication bias favors "statistically significant" results, and the methods also directly accommodate effect heterogeneity. Selection models have been established for decades in the statistics literature and are supported by user-friendly software, yet remain rarely reported in many disciplines. We use a previously published meta-analysis to demonstrate that selection models can yield insights that extend beyond those provided by funnel plot methods, suggesting the importance of establishing more comprehensive reporting practices for publication bias assessment.