Open Science in Education Sciences

A data science roadmap for open science organizations engaged in early-stage drug discovery

The Structural Genomics Consortium is an international open science research organization with a focus on accelerating early-stage drug discovery, namely hit discovery and optimization. We, as many others, believe that artificial intelligence (AI) is poised to be a main accelerator in the field. The question is then how to best benefit from recent advances in AI and how to generate, format and disseminate data to enable future breakthroughs in AI-guided drug discovery. We present here the recommendations of a working group composed of experts from both the public and private sectors. Robust data management requires precise ontologies and standardized vocabulary while a centralized database architecture across laboratories facilitates data integration into high-value datasets. Lab automation and opening electronic lab notebooks to data mining push the boundaries of data sharing and data modeling. Important considerations for building robust machine-learning models include transparent and reproducible data processing, choosing the most relevant data representation, defining the right training and test sets, and estimating prediction uncertainty. Beyond data-sharing, cloud-based computing can be harnessed to build and disseminate machine-learning models. Important vectors of acceleration for hit and chemical probe discovery will be (1) the real-time integration of experimental data generation and modeling workflows within design-make-test-analyze (DMTA) cycles openly, and at scale and (2) the adoption of a mindset where data scientists and experimentalists work as a unified team, and where data science is incorporated into the experimental design.

Using measurement alignment in research on adolescence involving multiple groups: A brief tutorial with R

Measurement alignment adjusts factor loadings and intercepts across different groups to achieve measurement invariance, which assumes the equal measurement model is validated across different groups. It should be achieved for validly conducting analysis and comparison in studies involving multiple groups, such as cross-cultural or cross-national studies. In this paper, I presented how to conduct measurement alignment with R. In addition to measurement alignment, I explained how to perform the Monte Carlo simulation to test the consistency and validity of alignment results and factor score calculation to facilitate further statistical analysis. A tutorial R code that implements all described procedures is freely shared via GitHub to inform readers who intend to use the alignment technique in their research projects.

Evaluation of Transparency and Openness Guidelines in Physical Therapist Journals

OBJECTIVE

The goals of this study were to evaluate the extent that physical therapist journals support open science research practices by adhering to the Transparency and Openness Promotion (TOP) guidelines and to assess the relationship between journal scores and their respective journal impact factor (JIF).

METHODS

Scimago, mapping studies, the National Library of Medicine, and journal author guidelines were searched to identify physical therapist journals for inclusion. Journals were graded on 10 standards (29 available total points) related to transparency with data, code, research materials, study design and analysis, preregistration of studies and statistical analyses, replication, and open science badges. The relationship between journal transparency and openness scores and their JIF was determined.

RESULTS

Thirty-five journals' author guidelines were assigned transparency and openness factor scores. The median score (interquartile range) across journals was 3.00 out of 29 (3.00) points (for all journals the scores ranged from 0 to 8). The 2 standards with the highest degree of implementation were design and analysis transparency (reporting guidelines) and study preregistration. No journals reported on code transparency, materials transparency, replication, and open science badges. TOP factor scores were a significant predictor of JIF scores.

CONCLUSION

There is low implementation of the TOP standards by physical therapist journals. TOP factor scores demonstrated predictive abilities for JIF scores. Policies from journals must improve to make open science practices the standard in research. Journals are in an influential position to guide practices that can improve the rigor of publication which, ultimately, enhances the evidence-based information used by physical therapists.

IMPACT

Transparent, open, and reproducible research will move the profession forward by improving the quality of research and increasing the confidence in results for implementation in clinical care.

Measuring CHAOS? Evaluating the short-form Confusion, Hubbub And Order Scale

The Confusion, Hubbub and Order Scale (CHAOS) - short form - is a survey tool intended to capture information about home environments. It is widely used in studies of child and adolescent development and psychopathology, particularly twin studies. The original long form of the scale comprised 15 items and was validated in a sample of infants in the 1980s. The short form of the scale was developed in the late 1990s and contains six items, including four from the original scale, and two new items. This short form has not been validated and is the focus of this study. We use five samples drawn from twin studies in Australia, the UK, and the USA, and examine measurement invariance of the CHAOS short-form. We first compare alternate confirmatory factor models for each group; we next test between-group configural, metric and scalar invariance; finally, we examine predictive validity of the scale under different conditions. We find evidence that a two-factor configuration of the six items is more appropriate than the commonly used one-factor model. Second, we find measurement non-invariance across groups at the metric invariance step, with items performing differently depending on the sample. We also find inconsistent results in tests of predictive validity using family-level socioeconomic status and academic achievement as criterion variables. The results caution the continued use of the short-form CHAOS in its current form and recommend future revisions and development of the scale for use in developmental research.

Low Research-Data Availability in Educational-Psychology Journals: No Indication of Effective Research-Data Policies

Research-data availability contributes to the transparency of the research process and the credibility of educational-psychology research and science in general. Recently, there have been many initiatives to increase the availability and quality of research data. Many research institutions have adopted research-data policies. This increased awareness might have raised the sharing of research data in empirical articles. To test this idea, we coded 1,242 publications from six educational-psychology journals and the psychological journal Cognition (as a baseline) published in 2018 and 2020. Research-data availability was low (3.85% compared with 62.74% in Cognition) but has increased from 0.32% (2018) to 7.16% (2020). However, neither the data-transparency level of the journal nor the existence of an official research-data policy on the level of the corresponding author’s institution was related to research-data availability. We discuss the consequences of these findings for institutional research-data-management processes.

A How-To Guide for Open-Science Practices in Special Education Research

Open-science reforms, which aim to increase credibility and access of research, have the potential to benefit the research base in special education, as well as practice and policy informed by that research base. Awareness of open science is increasing among special education researchers. However, relatively few researchers in the field have experience using multiple open-science practices, and few practical guidelines or resources have been tailored to special education researchers to support their exploration and adoption of open science. In this paper, we described and provided guidelines and resources for applying five core open-science practices-preregistration, registered reports, data sharing, materials sharing, and open-access publishing-in special education research.

Patterns in reporting and participant inclusion related to race and ethnicity in autism intervention literature: Data from a large-scale systematic review of evidence-based practices

There are marked racial and ethnic disparities in diagnosis and services for individuals on the autism spectrum, yet race and ethnicity are underreported and underexamined in autism research. The current study examines the reporting of race and ethnicity and the inclusion of participants across racial and ethnic groups in studies included in a large-scale systematic review of autism intervention research (1990–2017). Trained research assistants reviewed 1013 articles and extracted data on the reporting of race and ethnicity data and the inclusion of participants from different racial and ethnic categories from each article. Only 25% of the articles reported any data on race and ethnicity and reporting over time has slowly increased across the 28 years of the review. Descriptive statistics suggest that race and ethnicity reporting varied by study design, intervention, and outcomes. In studies with reported data, White participants had the highest rate of participation (64.8%), with a large gap between the next highest rates of participation, which were among Hispanic/Latino (9.4%), Black (7.7%), and Asian (6.4%) participants. The lack of reporting and the limited inclusion of participants across minoritized racial and ethnic groups are concerning and suggest a need to examine practices in autism research from planning to dissemination.

Open Accessibility in Education Research: Enhancing the Credibility, Equity, Impact, and Efficiency of Research

Openness is a foundational principle in science. Making the tools and products of scientific research openly accessible advances core aims and values of education researchers, such as the credibility, equity, impact, and efficiency of research. The digital revolution has expanded opportunities for providing greater access to research. In this article, we examine three open-science practices-open data and code, open materials, and open access-that education researchers can use to increase accessibility to the tools and products of research in the field. For each open-science practice, we discuss what the practice is and how it works, its primary benefits, some important limitations and challenges, and two thorny issues.