Ten simple rules on writing clean and reliable open-source scientific software

Best practices to evaluate the impact of biomedical research software-metric collection beyond citations

MOTIVATION

Software is vital for the advancement of biology and medicine. Impact evaluations of scientific software have primarily emphasized traditional citation metrics of associated papers, despite these metrics inadequately capturing the dynamic picture of impact and despite challenges with improper citation.

RESULTS

To understand how software developers evaluate their tools, we conducted a survey of participants in the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We found that although developers realize the value of more extensive metric collection, they find a lack of funding and time hindering. We also investigated software among this community for how often infrastructure that supports more nontraditional metrics were implemented and how this impacted rates of papers describing usage of the software. We found that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seemed to be associated with increased mention rates. Analysing more diverse metrics can enable developers to better understand user engagement, justify continued funding, identify novel use cases, pinpoint improvement areas, and ultimately amplify their software's impact. Challenges are associated, including distorted or misleading metrics, as well as ethical and security concerns. More attention to nuances involved in capturing impact across the spectrum of biomedical software is needed. For funders and developers, we outline guidance based on experience from our community. By considering how we evaluate software, we can empower developers to create tools that more effectively accelerate biological and medical research progress.

AVAILABILITY AND IMPLEMENTATION

More information about the analysis, as well as access to data and code is available at https://github.com/fhdsl/ITCR_Metrics_manuscript_website.

Ten simple rules to make computable knowledge shareable and reusable

Computable biomedical knowledge (CBK) is: "the result of an analytic and/or deliberative process about human health, or affecting human health, that is explicit, and therefore can be represented and reasned upon using logic, formal standards, and mathematical approaches." Representing biomedical knowledge in a machine-interpretable, computable form increases its ability to be discovered, accessed, understood, and deployed. Computable knowledge artifacts can greatly advance the potential for implementation, reproducibility, or extension of the knowledge by users, who may include practitioners, researchers, and learners. Enriching computable knowledge artifacts may help facilitate reuse and translation into practice. Following the examples of 10 Simple Rules papers for scientific code, software, and applications, we present 10 Simple Rules intended to make shared computable knowledge artifacts more useful and reusable. These rules are mainly for researchers and their teams who have decided that sharing their computable knowledge is important, who wish to go beyond simply describing results, algorithms, or models via traditional publication pathways, and who want to both make their research findings more accessible, and to help others use their computable knowledge. These rules are roughly organized into 3 categories: planning, engineering, and documentation. Finally, while many of the following examples are of computable knowledge in biomedical domains, these rules are generalizable to computable knowledge in any research domain.

Current status in spatiotemporal analysis of contrast-based perfusion MRI

In perfusion MRI, image voxels form a spatially organized network of systems, all exchanging indicator with their immediate neighbors. Yet the current paradigm for perfusion MRI analysis treats all voxels or regions-of-interest as isolated systems supplied by a single global source. This simplification not only leads to long-recognized systematic errors but also fails to leverage the embedded spatial structure within the data. Since the early 2000s, a variety of models and implementations have been proposed to analyze systems with between-voxel interactions. In general, this leads to large and connected numerical inverse problems that are intractible with conventional computational methods. With recent advances in machine learning, however, these approaches are becoming practically feasible, opening up the way for a paradigm shift in the approach to perfusion MRI. This paper seeks to review the work in spatiotemporal modelling of perfusion MRI using a coherent, harmonized nomenclature and notation, with clear physical definitions and assumptions. The aim is to introduce clarity in the state-of-the-art of this promising new approach to perfusion MRI, and help to identify gaps of knowledge and priorities for future research.

Implementing code review in the scientific workflow: Insights from ecology and evolutionary biology

Code review increases reliability and improves reproducibility of research. As such, code review is an inevitable step in software development and is common in fields such as computer science. However, despite its importance, code review is noticeably lacking in ecology and evolutionary biology. This is problematic as it facilitates the propagation of coding errors and a reduction in reproducibility and reliability of published results. To address this, we provide a detailed commentary on how to effectively review code, how to set up your project to enable this form of review and detail its possible implementation at several stages throughout the research process. This guide serves as a primer for code review, and adoption of the principles and advice here will go a long way in promoting more open, reliable, and transparent ecology and evolutionary biology.

Open-Source Machine Learning in Computational Chemistry

The field of computational chemistry has seen a significant increase in the integration of machine learning concepts and algorithms. In this Perspective, we surveyed 179 open-source software projects, with corresponding peer-reviewed papers published within the last 5 years, to better understand the topics within the field being investigated by machine learning approaches. For each project, we provide a short description, the link to the code, the accompanying license type, and whether the training data and resulting models are made publicly available. Based on those deposited in GitHub repositories, the most popular employed Python libraries are identified. We hope that this survey will serve as a resource to learn about machine learning or specific architectures thereof by identifying accessible codes with accompanying papers on a topic basis. To this end, we also include computational chemistry open-source software for generating training data and fundamental Python libraries for machine learning. Based on our observations and considering the three pillars of collaborative machine learning work, open data, open source (code), and open models, we provide some suggestions to the community.

Ten simple rules for providing bioinformatics support within a hospital

Bioinformatics has become a key aspect of the biomedical research programmes of many hospitals' scientific centres, and the establishment of bioinformatics facilities within hospitals has become a common practice worldwide. Bioinformaticians working in these facilities provide computational biology support to medical doctors and principal investigators who are daily dealing with data of patients to analyze. These bioinformatics analysts, although pivotal, usually do not receive formal training for this job. We therefore propose these ten simple rules to guide these bioinformaticians in their work: ten pieces of advice on how to provide bioinformatics support to medical doctors in hospitals. We believe these simple rules can help bioinformatics facility analysts in producing better scientific results and work in a serene and fruitful environment.

Evaluating LC-HRMS metabolomics data processing software using FAIR principles for research software

BACKGROUND

Liquid chromatography-high resolution mass spectrometry (LC-HRMS) is a popular approach for metabolomics data acquisition and requires many data processing software tools. The FAIR Principles - Findability, Accessibility, Interoperability, and Reusability - were proposed to promote open science and reusable data management, and to maximize the benefit obtained from contemporary and formal scholarly digital publishing. More recently, the FAIR principles were extended to include Research Software (FAIR4RS).

AIM OF REVIEW

This study facilitates open science in metabolomics by providing an implementation solution for adopting FAIR4RS in the LC-HRMS metabolomics data processing software. We believe our evaluation guidelines and results can help improve the FAIRness of research software.

KEY SCIENTIFIC CONCEPTS OF REVIEW

We evaluated 124 LC-HRMS metabolomics data processing software obtained from a systematic review and selected 61 software for detailed evaluation using FAIR4RS-related criteria, which were extracted from the literature along with internal discussions. We assigned each criterion one or more FAIR4RS categories through discussion. The minimum, median, and maximum percentages of criteria fulfillment of software were 21.6%, 47.7%, and 71.8%. Statistical analysis revealed no significant improvement in FAIRness over time. We identified four criteria covering multiple FAIR4RS categories but had a low %fulfillment: (1) No software had semantic annotation of key information; (2) only 6.3% of evaluated software were registered to Zenodo and received DOIs; (3) only 14.5% of selected software had official software containerization or virtual machine; (4) only 16.7% of evaluated software had a fully documented functions in code. According to the results, we discussed improvement strategies and future directions.

The in silico lab: Improving academic code using lessons from biology

"Good code" is often regarded as a nebulous, impractical ideal. Common best practices toward improving code quality can be inaccessible to those without a rigorous computer science or software engineering background, contributing to a gap between advancing scientific research and FAIR practices. We seek to equip researchers with the necessary background and context to tackle the challenge of improving code quality in computational biology research using analogies from biology to synthesize why certain best practices are critical for advancing computational research. Improving code quality requires active stewardship; we encourage researchers to deliberately adopt and share practices that ensure reusability, repeatability, and reproducibility.

Ten simple rules for using public biological data for your research

With an increasing amount of biological data available publicly, there is a need for a guide on how to successfully download and use this data. The 10 simple rules for using public biological data are: (1) use public data purposefully in your research; (2) evaluate data for your use case; (3) check data reuse requirements and embargoes; (4) be aware of ethics for data reuse; (5) plan for data storage and compute requirements; (6) know what you are downloading; (7) download programmatically and verify integrity; (8) properly cite data; (9) make reprocessed data and models Findable, Accessible, Interoperable, and Reusable (FAIR) and share; and (10) make pipelines and code FAIR and share. These rules are intended as a guide for researchers wanting to make use of available data and to increase data reuse and reproducibility.

Toward practical transparent verifiable and long-term reproducible research using Guix

Reproducibility crisis urge scientists to promote transparency which allows peers to draw same conclusions after performing identical steps from hypothesis to results. Growing resources are developed to open the access to methods, data and source codes. Still, the computational environment, an interface between data and source code running analyses, is not addressed. Environments are usually described with software and library names associated with version labels or provided as an opaque container image. This is not enough to describe the complexity of the dependencies on which they rely to operate on. We describe this issue and illustrate how open tools like Guix can be used by any scientist to share their environment and allow peers to reproduce it. Some steps of research might not be fully reproducible, but at least, transparency for computation is technically addressable. These tools should be considered by scientists willing to promote transparency and open science.

Considerations and challenges for sex-aware drug repurposing

Sex differences are essential factors in disease etiology and manifestation in many diseases such as cardiovascular disease, cancer, and neurodegeneration [33]. The biological influence of sex differences (including genomic, epigenetic, hormonal, immunological, and metabolic differences between males and females) and the lack of biomedical studies considering sex differences in their study design has led to several policies. For example, the National Institute of Health's (NIH) sex as a biological variable (SABV) and Sex and Gender Equity in Research (SAGER) policies to motivate researchers to consider sex differences [204]. However, drug repurposing, a promising alternative to traditional drug discovery by identifying novel uses for FDA-approved drugs, lacks sex-aware methods that can improve the identification of drugs that have sex-specific responses [7, 11, 14, 33]. Sex-aware drug repurposing methods either select drug candidates that are more efficacious in one sex or deprioritize drug candidates based on if they are predicted to cause a sex-bias adverse event (SBAE), unintended therapeutic effects that are more likely to occur in one sex. Computational drug repurposing methods are encouraging approaches to develop for sex-aware drug repurposing because they can prioritize sex-specific drug candidates or SBAEs at lower cost and time than traditional drug discovery. Sex-aware methods currently exist for clinical, genomic, and transcriptomic information [1, 7, 155]. They have not expanded to other data types, such as DNA variation, which has been beneficial in other drug repurposing methods that do not consider sex [114]. Additionally, some sex-aware methods suffer from poorer performance because a disproportionate number of male and female samples are available to train computational methods [7]. However, there is development potential for several different categories (i.e., data mining, ligand binding predictions, molecular associations, and networks). Low-dimensional representations of molecular association and network approaches are also especially promising candidates for future sex-aware drug repurposing methodologies because they reduce the multiple hypothesis testing burden and capture sex-specific variation better than the other methods [151, 159]. Here we review how sex influences drug response, the current state of drug repurposing including with respect to sex-bias drug response, and how model organism study design choices influence drug repurposing validation.