Transparency in Cognitive Training Meta-analyses: A Meta-review

Meta-analyses often present flexibility regarding their inclusion criteria, outcomes of interest, statistical analyses, and assessments of the primary studies. For this reason, it is necessary to transparently report all the information that could impact the results. In this meta-review, we aimed to assess the transparency of meta-analyses that examined the benefits of cognitive training, given the ongoing controversy that exists in this field. Ninety-seven meta-analytic reviews were included, which examined a wide range of populations with different clinical conditions and ages. Regarding the reporting, information about the search of the studies, screening procedure, or data collection was detailed by most reviews. However, authors usually failed to report other aspects such as the specific meta-analytic parameters, the formula used to compute the effect sizes, or the data from primary studies that were used to compute the effect sizes. Although some of these practices have improved over the years, others remained the same. Moreover, examining the eligibility criteria of the reviews revealed a great heterogeneity in aspects such as the training duration, age cut-offs, or study designs that were considered. Preregistered meta-analyses often specified poorly how they would deal with the multiplicity of data or assess publication bias in their protocols, and some contained non-disclosed deviations in their eligibility criteria or outcomes of interests. The findings shown here, although they do not question the benefits of cognitive training, illustrate important aspects that future reviews must consider.

Variation in global network properties across risk factors for adolescent internalizing symptoms: evidence of cumulative effects on structure and connectivity

BACKGROUND

Identifying adolescents at risk of internalizing problems is a key priority. However, studies have tended to consider such problems in simple ways using diagnoses, or item summaries. Network theory and methods instead allow for more complex interaction between symptoms. Two key hypotheses predict differences in global network properties for those at risk: altered structure and increased connectivity.

METHODS

The current study evaluated these hypotheses for nine risk factors (e.g. income deprivation and low parent/carer support) individually and cumulatively in a large sample of 12-15 year-olds (N = 34 564). Recursive partitioning and bootstrapped networks were used to evaluate structural and connectivity differences.

RESULTS

The pattern of network interactions was shown to be significantly different via recursive partitioning for all comparisons across risk-present/absent groups and levels of cumulative risk, except for income deprivation. However, the magnitude of differences appeared small. Most individual risk factors also showed relatively small effects for connectivity. Exceptions were noted for gender and sexual minority risk groups, as well as low parent/carer support, where larger effects were evident. A strong linear trend was observed between increasing cumulative risk exposure and connectivity.

CONCLUSIONS

A robust approach to considering the effect of risk exposure on global network properties was demonstrated. Results are consistent with the ideas that pathological states are associated with higher connectivity, and that the number of risks, regardless of their nature, is important. Gender/sexual minority status and low parent/carer support had the biggest individual impacts on connectivity, suggesting these are particularly important for identification and prevention.

Implementation of FAIR Practices in Computational Metabolomics Workflows-A Case Study

Scientific workflows facilitate the automation of data analysis tasks by integrating various software and tools executed in a particular order. To enable transparency and reusability in workflows, it is essential to implement the FAIR principles. Here, we describe our experiences implementing the FAIR principles for metabolomics workflows using the Metabolome Annotation Workflow (MAW) as a case study. MAW is specified using the Common Workflow Language (CWL), allowing for the subsequent execution of the workflow on different workflow engines. MAW is registered using a CWL description on WorkflowHub. During the submission process on WorkflowHub, a CWL description is used for packaging MAW using the Workflow RO-Crate profile, which includes metadata in Bioschemas. Researchers can use this narrative discussion as a guideline to commence using FAIR practices for their bioinformatics or cheminformatics workflows while incorporating necessary amendments specific to their research area.

Researcher degrees of freedom in statistical software contribute to unreliable results: A comparison of nonparametric analyses conducted in SPSS, SAS, Stata, and R

Researcher degrees of freedom can affect the results of hypothesis tests and consequently, the conclusions drawn from the data. Previous research has documented variability in accuracy, speed, and documentation of output across various statistical software packages. In the current investigation, we conducted Pearson's chi-square test of independence, Spearman's rank-ordered correlation, Kruskal-Wallis one-way analysis of variance, Wilcoxon Mann-Whitney U rank-sum tests, and Wilcoxon signed-rank tests, along with estimates of skewness and kurtosis, on large, medium, and small samples of real and simulated data in SPSS, SAS, Stata, and R and compared the results with those obtained through hand calculation using the raw computational formulas. Multiple inconsistencies were found in the results produced between statistical packages due to algorithmic variation, computational error, and statistical output. The most notable inconsistencies were due to algorithmic variations in the computation of Pearson's chi-square test conducted on 2 × 2 tables, where differences in p-values reported by different software packages ranged from .005 to .162, largely as a function of sample size. We discuss how such inconsistencies may influence the conclusions drawn from the results of statistical analyses depending on the statistical software used, and we urge researchers to analyze their data across multiple packages to check for inconsistencies and report details regarding the statistical procedure used for data analysis.

Testing Replicability and Generalizability of the Time on Task Effect

The time on task (ToT) effect describes the relationship of the time spent on a cognitive task and the probability of successful task completion. The effect has been shown to vary in size and direction across tests and even within tests, depending on the test taker and item characteristics. Specifically, investing more time has a positive effect on response accuracy for difficult items and low ability test-takers, but a negative effect for easy items and high ability test-takers. The present study sought to test the replicability of this result pattern of the ToT effect across samples independently drawn from the same populations of persons and items. Furthermore, its generalizability was tested in terms of differential correlations across ability tests. To this end, ToT effects were estimated for three different reasoning tests and one test measuring natural sciences knowledge in 10 comparable subsamples with a total N = 2640. Results for the subsamples were highly similar, demonstrating that ToT effects are estimated with sufficient reliability. Generally, faster answers tended to be more accurate, suggesting a relatively effortless processing style. However, with increasing item difficulty and decreasing person ability, the effect flipped to the opposite direction, i.e., higher accuracy with longer processing times. The within-task moderation of the ToT effect can be reconciled with an account on effortful processing or cognitive load. By contrast, the generalizability of the ToT effect across different tests was only moderate. Cross-test relations were stronger in relative terms if performance in the respective tasks was more strongly related. This suggests that individual differences in the ToT effect depend on test characteristics such as their reliabilities but also similarities and differences of their processing requirements.

Psychopathological networks: Theory, methods and practice

In recent years, network approaches to psychopathology have sparked much debate and have had a significant impact on how mental disorders are perceived in the field of clinical psychology. However, there are many important challenges in moving from theory to empirical research and clinical practice and vice versa. Therefore, in this article, we bring together different points of view on psychological networks by methodologists and clinicians to give a critical overview on these challenges, and to present an agenda for addressing these challenges. In contrast to previous reviews, we especially focus on methodological issues related to temporal networks. This includes topics such as selecting and assessing the quality of the nodes in the network, distinguishing between- and within-person effects in networks, relating items that are measured at different time scales, and dealing with changes in network structures. These issues are not only important for researchers using network models on empirical data, but also for clinicians, who are increasingly likely to encounter (person-specific) networks in the consulting room.

Polish Adaptation of the Pregnancy-Related Anxiety Questionnaire-Revised 2 for All Pregnant Women

Pregnancy-related anxiety (PrA) is a specific type of anxiety characteristic of the perinatal period. PrA can affect pregnancy and birth. However, no validated tool exists to measure PrA in Polish obstetric practice. The aim of this study was to translate the Pregnancy-Related Anxiety Questionnaire-Revised 2 (PRAQ-R2) into Polish and to evaluate its reliability and factorial and construct validity. This study was conducted in Poland as an online questionnaire in April 2020 and included 175 healthy women. To validate the PRAQ-R2, we used standardized tools for the measurement of general anxiety: the modified Visual Analog Scale (VAS), the Ten-Item Personality Inventory (TIPI), and the Hospital Anxiety and Depression Scale (HADS). Scale reliability was assessed using Cronbach's alpha. Concurrent validity was evaluated by calculating Spearman's rho correlation coefficients. Statistical analyses were performed using R ver. 4.0.2. Values for comparative fit index >0.90, Tucker-Lewis index >0.90, and root mean square error of approximation <0.08 indicated acceptable model fit, confirming the reliability of the three-factor structure of the translation. The subscales and total scores had good consistency (α > 0.7), and convergent validity was demonstrated. The PRAQ-R2 as translated into Polish represents the first validated tool in Poland to measure PrA for all pregnant women.

A meta-review of transparency and reproducibility-related reporting practices in published meta-analyses on clinical psychological interventions (2000-2020)

Meta-analysis is a powerful and important tool to synthesize the literature about a research topic. Like other kinds of research, meta-analyses must be reproducible to be compliant with the principles of the scientific method. Furthermore, reproducible meta-analyses can be easily updated with new data and reanalysed applying new and more refined analysis techniques. We attempted to empirically assess the prevalence of transparency and reproducibility-related reporting practices in published meta-analyses from clinical psychology by examining a random sample of 100 meta-analyses. Our purpose was to identify the key points that could be improved, with the aim of providing some recommendations for carrying out reproducible meta-analyses. We conducted a meta-review of meta-analyses of psychological interventions published between 2000 and 2020. We searched PubMed, PsycInfo and Web of Science databases. A structured coding form to assess transparency indicators was created based on previous studies and existing meta-analysis guidelines. We found major issues concerning: completely reproducible search procedures report, specification of the exact method to compute effect sizes, choice of weighting factors and estimators, lack of availability of the raw statistics used to compute the effect size and of interoperability of available data, and practically total absence of analysis script code sharing. Based on our findings, we conclude with recommendations intended to improve the transparency, openness, and reproducibility-related reporting practices of meta-analyses in clinical psychology and related areas.

Internalizing symptoms, well-being, and correlates in adolescence: A multiverse exploration via cross-lagged panel network models

Internalizing symptoms are the most prevalent mental health problem in adolescents, with sharp increases seen, particularly for girls, and evidence that young people today report more problems than previous generations. It is therefore critical to measure and monitor these states on a large scale and consider correlates. We used novel panel network methodology to explore relationships between internalizing symptoms, well-being, and inter/intrapersonal indicators. A multiverse design was used with 32 conditions to consider the stability of results across arbitrary researcher decisions in a large community sample over three years (N = 15,843, aged 11-12 at Time 1). Networks were consistently similar for girls and boys. Stable trait-like effects within anxiety, attentional, and social indicators were found. Within-person networks were densely connected and suggested mental health and inter/intrapersonal correlates related to one another in similar complex ways. The multiverse design suggested the particular operationalization of items can substantially influence conclusions. Nevertheless, indicators such as thinking clearly, unhappiness, dealing with stress, and worry showed more consistent centrality, suggesting these indicators may play particularly important roles in the development of mental health in adolescence.

Leveraging Containers for Reproducible Psychological Research

Containers have become increasingly popular in computing and software engineering and are gaining traction in scientific research. They allow packaging up all code and dependencies to ensure that analyses run reliably across a range of operating systems and software versions. Despite being a crucial component for reproducible science, containerization has yet to become mainstream in psychology. In this tutorial, we describe the logic behind containers, what they are, and the practical problems they can solve. We walk the reader through the implementation of containerization within a research workflow with examples using Docker and R. Specifically, we describe how to use existing containers, build personalized containers, and share containers alongside publications. We provide a worked example that includes all steps required to set up a container for a research project and can easily be adapted and extended. We conclude with a discussion of the possibilities afforded by the large-scale adoption of containerization, especially in the context of cumulative, open science, toward a more efficient and inclusive research ecosystem.

Open Science in Education Sciences

The Open Science movement has gained considerable traction in the last decade. The Open Science movement tries to increase trust in research results and open the access to all elements of a research project to the public. Central to these goals, Open Science has promoted five critical tenets: Open Data, Open Analysis, Open Materials, Preregistration, and Open Access. All Open Science elements can be thought of as extensions to the traditional way of achieving openness in science, which has been scientific publication of research outcomes in journals or books. Open Science in education sciences, however, has the potential to be much more than a safeguard against questionable research. Open Science in education science provides opportunities to (a) increase the transparency and therefore replicability of research and (b) develop and answer research questions about individuals with learning disabilities and learning difficulties that were previously impossible to answer due to complexities in data analysis methods. We will provide overviews of the main tenets of Open Science (i.e., Open Data, Open Analysis, Open Materials, Preregistration, and Open Access), show how they are in line with grant funding agencies' expectations for rigorous research processes, and present resources on best practices for each of the tenets.

Text Preprocessing for Text Mining in Organizational Research: Review and Recommendations

Recent advances in text mining have provided new methods for capitalizing on the voluminous natural language text data created by organizations, their employees, and their customers. Although often overlooked, decisions made during text preprocessing affect whether the content and/or style of language are captured, the statistical power of subsequent analyses, and the validity of insights derived from text mining. Past methodological articles have described the general process of obtaining and analyzing text data, but recommendations for preprocessing text data were inconsistent. Furthermore, primary studies use and report different preprocessing techniques. To address this, we conduct two complementary reviews of computational linguistics and organizational text mining research to provide empirically grounded text preprocessing decision-making recommendations that account for the type of text mining conducted (i.e., open or closed vocabulary), the research question under investigation, and the data set’s characteristics (i.e., corpus size and average document length). Notably, deviations from these recommendations will be appropriate and, at times, necessary due to the unique characteristics of one’s text data. We also provide recommendations for reporting text mining to promote transparency and reproducibility.

ACTman: Automated preprocessing and analysis of actigraphy data

OBJECTIVES

To introduce a novel software-library called Actigraphy Manager (ACTman) which automates labor-intensive actigraphy data preprocessing and analyses steps while improving transparency, reproducibility, and scalability over software suites traditionally used in actigraphy research practice.

DESIGN

Descriptive.

METHODS

Use cases are described for performing a common actigraphy task in ACTman and alternative actigraphy software. Important inefficiencies in actigraphy workflow are identified and their consequences are described. We explain how these hinder the feasibility of conducting studies with large groups of athletes and/or longer data collection periods. Thereafter, the information flow through the ACTman software is described and we explain how it alleviates aforementioned inefficiencies. Furthermore, transparency, reproducibility, and scalability issues of commonly used actigraphy software packages are discussed and compared with the ACTman package.

RESULTS

It is shown that from an end-user perspective ACTman offers a compact workflow as it automates many preprocessing and analysis steps that otherwise have to be performed manually. When considering transparency, reproducibility, and scalability the design of the ACTman software is found to outperform proprietary and open-source actigraphy software suites. As such, ACTman alleviates important bottlenecks within actigraphy research practice.

CONCLUSIONS

ACTman facilitates the current transition towards larger datasets containing data of multiple athletes by automating labor-intensive preprocessing and analyses steps within actigraphy research. Furthermore, ACTman offers many features which enhance user-convenience and analysis customization, such as moving window functionality and period selection options. ACTman is open-source and thus fully verifiable, in contrast with many proprietary software packages which remain a black box for researchers.