Reproducible Research: A Retrospective

Ledged Beam Walking Test Automatic Tracker: Artificial intelligence-based functional evaluation in a stroke model

The quantitative evaluation of motor function in experimental stroke models is essential for the preclinical assessment of new therapeutic strategies that can be transferred to clinical research; however, conventional assessment tests are hampered by the evaluator's subjectivity. We present an artificial intelligence-based system for the automatic, accurate, and objective analysis of target parameters evaluated by the ledged beam walking test, which offers higher sensitivity than the current methodology based on manual and visual counting. This system employs a residual deep network model, trained with DeepLabCut (DLC) to extract target paretic hindlimb coordinates, which are categorized to provide a ratio measurement of the animal's neurological deficit. The results correlate with the measurements performed by a professional observer and have greater reproducibility, easing the analysis of motor deficits and providing a reliable and useful tool applicable to other diseases causing motor deficits.

COVID-19 vaccination strategies in Africa: A scoping review of the use of mathematical models to inform policy

OBJECTIVE

Mathematical models are vital tools to understand transmission dynamics and assess the impact of interventions to mitigate COVID-19. However, historically, their use in Africa has been limited. In this scoping review, we assess how mathematical models were used to study COVID-19 vaccination to potentially inform pandemic planning and response in Africa.

METHODS

We searched six electronic databases: MEDLINE, Embase, Web of Science, Global Health, MathSciNet and Africa-Wide NiPAD, using keywords to identify articles focused on the use of mathematical modelling studies of COVID-19 vaccination in Africa that were published as of October 2022. We extracted the details on the country, author affiliation, characteristics of models, policy intent and heterogeneity factors. We assessed quality using 21-point scale criteria on model characteristics and content of the studies.

RESULTS

The literature search yielded 462 articles, of which 32 were included based on the eligibility criteria. Nineteen (59%) studies had a first author affiliated with an African country. Of the 32 included studies, 30 (94%) were compartmental models. By country, most studies were about or included South Africa (n = 12, 37%), followed by Morocco (n = 6, 19%) and Ethiopia (n = 5, 16%). Most studies (n = 19, 59%) assessed the impact of increasing vaccination coverage on COVID-19 burden. Half (n = 16, 50%) had policy intent: prioritising or selecting interventions, pandemic planning and response, vaccine distribution and optimisation strategies and understanding transmission dynamics of COVID-19. Fourteen studies (44%) were of medium quality and eight (25%) were of high quality.

CONCLUSIONS

While decision-makers could draw vital insights from the evidence generated from mathematical modelling to inform policy, we found that there was limited use of such models exploring vaccination impacts for COVID-19 in Africa. The disparity can be addressed by scaling up mathematical modelling training, increasing collaborative opportunities between modellers and policymakers, and increasing access to funding.

Introduction of the Capsules environment to support further growth of the SBGrid structural biology software collection

The expansive scientific software ecosystem, characterized by millions of titles across various platforms and formats, poses significant challenges in maintaining reproducibility and provenance in scientific research. The diversity of independently developed applications, evolving versions and heterogeneous components highlights the need for rigorous methodologies to navigate these complexities. In response to these challenges, the SBGrid team builds, installs and configures over 530 specialized software applications for use in the on-premises and cloud-based computing environments of SBGrid Consortium members. To address the intricacies of supporting this diverse application collection, the team has developed the Capsule Software Execution Environment, generally referred to as Capsules. Capsules rely on a collection of programmatically generated bash scripts that work together to isolate the runtime environment of one application from all other applications, thereby providing a transparent cross-platform solution without requiring specialized tools or elevated account privileges for researchers. Capsules facilitate modular, secure software distribution while maintaining a centralized, conflict-free environment. The SBGrid platform, which combines Capsules with the SBGrid collection of structural biology applications, aligns with FAIR goals by enhancing the findability, accessibility, interoperability and reusability of scientific software, ensuring seamless functionality across diverse computing environments. Its adaptability enables application beyond structural biology into other scientific fields.

A guide for social science journal editors on easing into open science

Journal editors have a large amount of power to advance open science in their respective fields by incentivising and mandating open policies and practices at their journals. The Data PASS Journal Editors Discussion Interface (JEDI, an online community for social science journal editors: www.dpjedi.org ) has collated several resources on embedding open science in journal editing ( www.dpjedi.org/resources ). However, it can be overwhelming as an editor new to open science practices to know where to start. For this reason, we created a guide for journal editors on how to get started with open science. The guide outlines steps that editors can take to implement open policies and practices within their journal, and goes through the what, why, how, and worries of each policy and practice. This manuscript introduces and summarizes the guide (full guide: https://doi.org/10.31219/osf.io/hstcx ).

Principles for Constructing DNA Barcode Reference Libraries

All DNA barcode methods rely on reference sequences linked to well-curated voucher specimens. Definitions for and locations of DNA barcode reference libraries are not standardized, and vary throughout the literature. Standardizing, and centralizing reference specimens would provide an unambiguous source, analogous to reference genomes, to reproduce identifications and improve a library. This chapter proposes a working definition of a DNA barcode reference library, consistent with DNA barcode data standards, along with principles and methods to consider when producing or using such a library. These methods allow explicit traceback to sequence-sources which elevate the value of voucher specimens, and create a potential for community curation.

The Quartet Data Portal: integration of community-wide resources for multiomics quality control

The Quartet Data Portal facilitates community access to well-characterized reference materials, reference datasets, and related resources established based on a family of four individuals with identical twins from the Quartet Project. Users can request DNA, RNA, protein, and metabolite reference materials, as well as datasets generated across omics, platforms, labs, protocols, and batches. Reproducible analysis tools allow for objective performance assessment of user-submitted data, while interactive visualization tools support rapid exploration of reference datasets. A closed-loop "distribution-collection-evaluation-integration" workflow enables updates and integration of community-contributed multiomics data. Ultimately, this portal helps promote the advancement of reference datasets and multiomics quality control.

Data Management: The First Step in Reproducible Research

Reproducibility is a preferred aim in any scientific research, including occupational health research. Datamanagement is an important and essential step in marching towards reproducibility. A good datamanagement helps us stay organized, improve transparency, quality and fosters collaboration. Here we discuss how to organize and prepare for data management, how data management facilitates interoperability and accessibility, followed by storing and dissemination of data. We wrap up by providing pointers on what needs to be included in the data management plans.

Machine Learning in Laboratory Medicine: Recommendations of the IFCC Working Group

BACKGROUND

Machine learning (ML) has been applied to an increasing number of predictive problems in laboratory medicine, and published work to date suggests that it has tremendous potential for clinical applications. However, a number of groups have noted the potential pitfalls associated with this work, particularly if certain details of the development and validation pipelines are not carefully controlled.

METHODS

To address these pitfalls and other specific challenges when applying machine learning in a laboratory medicine setting, a working group of the International Federation for Clinical Chemistry and Laboratory Medicine was convened to provide a guidance document for this domain.

RESULTS

This manuscript represents consensus recommendations for best practices from that committee, with the goal of improving the quality of developed and published ML models designed for use in clinical laboratories.

CONCLUSIONS

The committee believes that implementation of these best practices will improve the quality and reproducibility of machine learning utilized in laboratory medicine.

SUMMARY

We have provided our consensus assessment of a number of important practices that are required to ensure that valid, reproducible machine learning (ML) models can be applied to address operational and diagnostic questions in the clinical laboratory. These practices span all phases of model development, from problem formulation through predictive implementation. Although it is not possible to exhaustively discuss every potential pitfall in ML workflows, we believe that our current guidelines capture best practices for avoiding the most common and potentially dangerous errors in this important emerging field.

The impact of commercial health datasets on medical research and health-care algorithms

As the health-care industry emerges into a new era of digital health driven by cloud data storage, distributed computing, and machine learning, health-care data have become a premium commodity with value for private and public entities. Current frameworks of health data collection and distribution, whether from industry, academia, or government institutions, are imperfect and do not allow researchers to leverage the full potential of downstream analytical efforts. In this Health Policy paper, we review the current landscape of commercial health data vendors, with special emphasis on the sources of their data, challenges associated with data reproducibility and generalisability, and ethical considerations for data vending. We argue for sustainable approaches to curating open-source health data to enable global populations to be included in the biomedical research community. However, to fully implement these approaches, key stakeholders should come together to make health-care datasets increasingly accessible, inclusive, and representative, while balancing the privacy and rights of individuals whose data are being collected.

Reproducible variability: assessing investigator discordance across 9 research teams attempting to reproduce the same observational study

OBJECTIVE

Observational studies can impact patient care but must be robust and reproducible. Nonreproducibility is primarily caused by unclear reporting of design choices and analytic procedures. This study aimed to: (1) assess how the study logic described in an observational study could be interpreted by independent researchers and (2) quantify the impact of interpretations' variability on patient characteristics.

MATERIALS AND METHODS

Nine teams of highly qualified researchers reproduced a cohort from a study by Albogami et al. The teams were provided the clinical codes and access to the tools to create cohort definitions such that the only variable part was their logic choices. We executed teams' cohort definitions against the database and compared the number of subjects, patient overlap, and patient characteristics.

RESULTS

On average, the teams' interpretations fully aligned with the master implementation in 4 out of 10 inclusion criteria with at least 4 deviations per team. Cohorts' size varied from one-third of the master cohort size to 10 times the cohort size (2159-63 619 subjects compared to 6196 subjects). Median agreement was 9.4% (interquartile range 15.3-16.2%). The teams' cohorts significantly differed from the master implementation by at least 2 baseline characteristics, and most of the teams differed by at least 5.

CONCLUSIONS

Independent research teams attempting to reproduce the study based on its free-text description alone produce different implementations that vary in the population size and composition. Sharing analytical code supported by a common data model and open-source tools allows reproducing a study unambiguously thereby preserving initial design choices.

A simple kit to use computational notebooks for more openness, reproducibility, and productivity in research

The ubiquitous use of computational work for data generation, processing, and modeling increased the importance of digital documentation in improving research quality and impact. Computational notebooks are files that contain descriptive text, as well as code and its outputs, in a single, dynamic, and visually appealing file that is easier to understand by nonspecialists. Traditionally used by data scientists when producing reports and informing decision-making, the use of this tool in research publication is not common, despite its potential to increase research impact and quality. For a single study, the content of such documentation partially overlaps with that of classical lab notebooks and that of the scientific manuscript reporting the study. Therefore, to minimize the amount of work required to manage all the files related to these contents and optimize their production, we present a starter kit to facilitate the implementation of computational notebooks in the research process, including publication. The kit contains the template of a computational notebook integrated into a research project that employs R, Python, or Julia. Using examples of ecological studies, we show how computational notebooks also foster the implementation of principles of Open Science, such as reproducibility and traceability. The kit is designed for beginners, but at the end we present practices that can be gradually implemented to develop a fully digital research workflow. Our hope is that such minimalist yet effective starter kit will encourage researchers to adopt this practice in their workflow, regardless of their computational background.

The Open Science movement has been gaining track in recent years by reinforcing the bigger impact that collaborative research has: the more publicly available research there is, the easier it is to trust and build upon it. A key feature of effectively “available” and reusable research is being well documented, so it can be easily understood by those who need it. However, well documenting scientific work can be a daunting task and scientists may fall prey to workloads that are too heavy and possibly inefficient, if they are not familiar with the tools available for it. At the same time, since most research is conducted with at least one computational element (e.g., data analysis or storage of data in digital databases), the time is ripe to learn methods of documenting computational work. In this guide, we provide a minimal yet versatile set up to help scientists conduct and document their research in a more understandable, shareable, and impactful way.

Success4life Youth Empowerment for Promoting Well-being and Boosting Mental Health: Protocol for an Experimental Study

BACKGROUND

There is an increasingly alarming worsening of mental health among the youth. There remain significant unmet needs for developing innovative, evidence-based technology-enhanced, positive psychology interventions (PPIs) all-inclusive in targeting psychological distress and risk factors related to high-risk behavior commonly encountered in adolescents.

OBJECTIVE

We aim to assess the effectiveness of a hybrid (incorporating both synchronous and asynchronous learning) and holistic (targeting social and emotional learning and tackling risk factors unique for this age group) PPI, "success4life youth empowerment," in improving well-being in the youth.

METHODS

Students' well-being will be assessed by the 5-item World Health Organization Well-Being Index, and hope will be assessed by the 6-item Children's Hope Scale at week 0, week 8, and week 10, month 6, and month 12. Any improvement in well-being and hope will be measured, estimating the difference in postintervention (week 8 and week 10) and preintervention (week 0) scores by determining the P value and effect size using appropriate statistical tests.

RESULTS

This study includes 2 phases: pilot phase 1, delivered by the creators of the succcess4life youth empowerment modules and platform, and phase 2, which will consist of the estimation of scalability through the recruitment of trainers. We hope to start student recruitment by 2022 and aim to complete the results for phase 1 pilot testing by 2023.

CONCLUSIONS

We anticipate that a primarily web-based, 10-week holistic PPI can support improvement in the mental wellness of the youth and has the potential for effective scalability.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/38463.

Objectively measuring the association between the built environment and physical activity: a systematic review and reporting framework

Background

Objective measures of built environment and physical activity provide the opportunity to directly compare their relationship across different populations and spatial contexts. This systematic review synthesises the current body of knowledge and knowledge gaps around the impact of objectively measured built environment metrics on physical activity levels in adults (≥ 18 years). Additionally, this review aims to address the need for improved quality of methodological reporting to evaluate studies and improve inter-study comparability though the creation of a reporting framework.

Methods

A systematic search of the literature was conducted following the PRISMA guidelines. After abstract and full-text screening, 94 studies were included in the final review. Results were synthesised using an association matrix to show overall association between built environment and physical activity variables. Finally, the new PERFORM (’Physical and Environmental Reporting Framework for Objectively Recorded Measures’) checklist was created and applied to the included studies rating them on their reporting quality across four key areas: study design and characteristics, built environment exposures, physical activity metrics, and the association between built environment and physical activity.

Results

Studies came from 21 countries and ranged from two days to six years in duration. Accelerometers and using geographic information system (GIS) to define the spatial extent of exposure around a pre-defined geocoded location were the most popular tools to capture physical activity and built environment respectively. Ethnicity and socio-economic status of participants were generally poorly reported. Moderate-to-vigorous physical activity (MVPA) was the most common metric of physical activity used followed by walking. Commonly investigated elements of the built environment included walkability, access to parks and green space. Areas where there was a strong body of evidence for a positive or negative association between the built environment and physical activity were identified. The new PERFORM checklist was devised and poorly reported areas identified, included poor reporting of built environment data sources and poor justification of method choice.

Conclusions

This systematic review highlights key gaps in studies objectively measuring the built environment and physical activity both in terms of the breadth and quality of reporting. Broadening the variety measures of the built environment and physical activity across different demographic groups and spatial areas will grow the body and quality of evidence around built environment effect on activity behaviour. Whilst following the PERFORM reporting guidance will ensure the high quality, reproducibility, and comparability of future research.

Ten simple rules for maximizing the recommendations of the NIH data management and sharing plan

The National Institutes of Health (NIH) Policy for Data Management and Sharing (DMS Policy) recognizes the NIH’s role as a key steward of United States biomedical research and information and seeks to enhance that stewardship through systematic recommendations for the preservation and sharing of research data generated by funded projects. The policy is effective as of January 2023. The recommendations include a requirement for the submission of a Data Management and Sharing Plan (DMSP) with funding applications, and while no strict template was provided, the NIH has released supplemental draft guidance on elements to consider when developing a plan. This article provides 10 key recommendations for creating a DMSP that is both maximally compliant and effective.

BioVisReport: A Markdown-based lightweight website builder for reproducible and interactive visualization of results from peer-reviewed publications

Interactive visualization is an effective way to promote the reproducibility of results presented in biomedical publications and to facilitate additional exploration of the reported data. However, there is a lack of convenient tools that balance reproducibility with ease of use. To address this problem, we develop BioVisReport, a lightweight solution for the rapid generation of an interactive website based on a user-defined Markdown file, which acts as a text markup language without requiring users to master complex syntax and allows them to preview the results in real-time. Interactive websites generated by the tool can help readers conveniently reproduce research findings and perform further in-depth analyses beyond those reported in the original peer-reviewed publications. Currently, BioVisReport offers 17 basic types of plots for visualizing published data. In addition, the extensibility of BioVisReport supports flexible integration of user-developed Python plugins with multiple programming languages. BioVisReport is freely available at https://biovis.report/.

An Open-Access Data Platform: Global Nutrition and Health Atlas (GNHA)

The rapid development of nutrition science is embracing digital transformation to generate large amounts of data. Precision nutrition and "Big Data" place increasing demand for data repositories and visualization, which enhances the digital transformation. We defined the need for an integrated nutrition data platform as a web-based platform that can collect, store, track, analyze, monitor, and visually display key metrics in nutrition and health while allowing users to interact with visuals and download data provided in the platform. Interactive dashboards create new opportunities for scholars and practitioners to generate and test hypotheses. We present the development and implementation of the Global Nutrition and Health Atlas (GNHA; https://sites.tufts.edu/gnha/), an open-access online platform covering nutrition and health data with 26 themes and 500+ indicators from 190+ countries up to 30 y. We view GNHA as an interactive tool aiming to share information and perspectives and foster collaborations and innovations.

FA-nf: A Functional Annotation Pipeline for Proteins from Non-Model Organisms Implemented in Nextflow

Functional annotation allows adding biologically relevant information to predicted features in genomic sequences, and it is, therefore, an important procedure of any de novo genome sequencing project. It is also useful for proofreading and improving gene structural annotation. Here, we introduce FA-nf, a pipeline implemented in Nextflow, a versatile computational workflow management engine. The pipeline integrates different annotation approaches, such as NCBI BLAST+, DIAMOND, InterProScan, and KEGG. It starts from a protein sequence FASTA file and, optionally, a structural annotation file in GFF format, and produces several files, such as GO assignments, output summaries of the abovementioned programs and final annotation reports. The pipeline can be broken easily into smaller processes for the purpose of parallelization and easily deployed in a Linux computational environment, thanks to software containerization, thus helping to ensure full reproducibility.

The Reliability and Sensitivity of Change of Direction Deficit and Its Association with Linear Sprint Speed in Prepubertal Male Soccer Players

BACKGROUND

This study aimed to examine the reliability and sensitivity of a change of direction deficit (CoDD) and to establish its relationship with linear sprint speed.

METHODS

In total, 89 prepubertal male soccer players participated in this study (age = 11.7 ± 1.2 years, maturity offset = -2.4 ± 1.0). Participants performed the 505 CoD speed test and the 20 m linear sprint speed test with a split interval at 5 m and 10 m. The CoDD was calculated as the mean 505 CoD time-the mean 10 to 20 m time interval. To evaluate the reliability of CoDD, the 505 CoD speed test, and 20 m linear sprint speed were performed twice, one week apart. The sensitivity of CoDD was identified by comparing the values of the typical error of measurement (TEM) and smallest worthwhile change (SWC).

RESULTS

Results of the reliability analysis indicated an intraclass correlation coefficient (ICC3.1) < 0.50 (0.47) and a TEM expressed as the coefficient of variation > 5% (10.55%). The sensitivity analysis showed that the ability of the CoDD measure to detect small performance changes is "marginal" (TEM (0.12) > SWC0.2 (0.04)). However, good absolute and relative reliability were observed for the 505 CoD speed test (ICC3.1 = 0.75; TEM < 5%). Alike CoDD, the ability of the 505 CoD speed test to detect small performance changes was rated as "marginal" (TEM (0.07 s) > SWC0.2 (0.04 s)). The CoDD revealed a large association with the 505 CoD speed test (r = 0.71). However, non-significant associations were detected between the CoDD and 5 m, 10 m, and 20 m linear sprint speed intervals (r = 0.10 to 0.16, all p > 0.05). Likewise, non-significant correlations between the 505 CoD speed test and 5 m, 10 m, and 20 m linear sprint speed intervals were observed (r = 0.14 to 0.20, all p > 0.05).

CONCLUSIONS

The CoDD displayed poor reliability and limited ability to detect small changes in performance in prepubertal male soccer players. Due to its limited practical utility, practitioners are advised not to consider CoDD scores during the assessment of prepubertal male soccer players.