Commentary on the integration of model sharing and reproducibility analysis to scholarly publishing workflow in computational biomechanics

Reproducibility in modeling and simulation of the knee: Academic, industry, and regulatory perspectives

Stakeholders in the modeling and simulation (M&S) community organized a workshop at the 2019 Annual Meeting of the Orthopaedic Research Society (ORS) entitled "Reproducibility in Modeling and Simulation of the Knee: Academic, Industry, and Regulatory Perspectives." The goal was to discuss efforts among these stakeholders to address irreproducibility in M&S focusing on the knee joint. An academic representative from a leading orthopedic hospital in the United States described a multi-institutional, open effort funded by the National Institutes of Health to assess model reproducibility in computational knee biomechanics. A regulatory representative from the United States Food and Drug Administration indicated the necessity of standards for reproducibility to increase utility of M&S in the regulatory setting. An industry representative from a major orthopedic implant company emphasized improving reproducibility by addressing indeterminacy in personalized modeling through sensitivity analyses, thereby enhancing preclinical evaluation of joint replacement technology. Thought leaders in the M&S community stressed the importance of data sharing to minimize duplication of efforts. A survey comprised 103 attendees revealed strong support for the workshop and for increasing emphasis on computational modeling at future ORS meetings. Nearly all survey respondents (97%) considered reproducibility to be an important issue. Almost half of respondents (45%) tried and failed to reproduce the work of others. Two-thirds of respondents (67%) declared that individual laboratories are most responsible for ensuring reproducible research whereas 44% thought that journals are most responsible. Thought leaders and survey respondents emphasized that computational models must be reproducible and credible to advance knee M&S.

Assessment of reporting practices and reproducibility potential of a cohort of published studies in computational knee biomechanics

Reproducible research serves as a pillar of the scientific method and is a foundation for scientific advancement. However, estimates for irreproducibility of preclinical science range from 75% to 90%. The importance of reproducible science has not been assessed in the context of mechanics-based modeling of human joints such as the knee, despite this being an area that has seen dramatic growth. Framed in the context of five experienced teams currently documenting knee modeling procedures, the aim of this study was to evaluate reporting and the perceived potential for reproducibility across studies the teams viewed as important contributions to the literature. A cohort of studies was selected by polling, which resulted in an assessment of nine studies as opposed to a broader analysis across the literature. Using a published checklist for reporting of modeling features, the cohort was evaluated for both "reporting" and their potential to be "reproduced," which was delineated into six major modeling categories and three subcategories. Logistic regression analysis revealed that for individual modeling categories, the proportion of "reported" occurrences ranged from 0.31, 95% confidence interval (CI) [0.23, 0.41] to 0.77, 95% CI: [0.68, 0.86]. The proportion of whether a category was perceived as "reproducible" ranged from 0.22, 95% CI: [0.15, 0.31] to 0.44, 95% CI: [0.35, 0.55]. The relatively low ratios highlight an opportunity to improve reporting and reproducibility of knee modeling studies. Ongoing efforts, including our findings, contribute to a dialogue that facilitates adoption of practices that provide both credibility and translation possibilities.

Open Knee(s): A Free and Open Source Library of Specimen-Specific Models and Related Digital Assets for Finite Element Analysis of the Knee Joint

There is a growing interest in the use of virtual representations of the knee for musculoskeletal research and clinical decision making, and to generate digital evidence for design and regulation of implants. Accessibility to previously developed models and related digital assets can dramatically reduce barriers to entry to conduct simulation-based studies of the knee joint and therefore help accelerate scientific discovery and clinical innovations. Development of models for finite element analysis is a demanding process that is both time consuming and resource intensive. It necessitates expertise to transform raw data to reliable virtual representations. Modeling and simulation workflow has many processes such as image segmentation, surface geometry generation, mesh generation and finally, creation of a finite element representation with relevant loading and boundary conditions. The outcome of the workflow is not only the end-point knee model but also many other digital by-products. When all of these data, derivate assets, and tools are freely and openly accessible, researchers can bypass some or all the steps required to build models and focus on using them to address their research goals. With provenance to specimen-specific anatomical and mechanical data and traceability of digital assets throughout the whole lifecycle of the model, reproducibility and credibility of the modeling practice can be established. The objective of this study is to disseminate Open Knee(s), a cohort of eight knee models (and relevant digital assets) for finite element analysis, that are based on comprehensive specimen-specific imaging data. In addition, the models and by-products of modeling workflows are described along with model development strategies and tools. Passive flexion served as a test simulation case, demonstrating an end-user application. Potential roadmaps for reuse of Open Knee(s) are also discussed.

OpenColab project: OpenSim in Google colaboratory to explore biomechanics on the web

OpenSim is an open-source biomechanical package with a variety of applications. It is available for many users with bindings in MATLAB, Python, and Java via its application programming interfaces (APIs). Although the developers described well the OpenSim installation on different operating systems (Windows, Mac, and Linux), it is time-consuming and complex since each operating system requires a different configuration. This project aims to demystify the development of neuro-musculoskeletal modeling in OpenSim with zero configuration on any operating system for installation (thus cross-platform), easy to share models while accessing free graphical processing units (GPUs) on a web-based platform of Google Colab. To achieve this, OpenColab was developed where OpenSim source code was used to build a Conda package that can be installed on the Google Colab with only one block of code in less than 7 min. To use OpenColab, one requires a connection to the internet and a Gmail account. Moreover, OpenColab accesses vast libraries of machine learning methods available within free Google products, e.g. TensorFlow. Next, we performed an inverse problem in biomechanics and compared OpenColab results with OpenSim graphical user interface (GUI) for validation. The outcomes of OpenColab and GUI matched well (r≥0.82). OpenColab takes advantage of the zero-configuration of cloud-based platforms, accesses GPUs, and enables users to share and reproduce modeling approaches for further validation, innovative online training, and research applications. Step-by-step installation processes and examples are available at: https://simtk.org/projects/opencolab.

Clinical Impact of Computational Heart Valve Models

This paper provides a review of engineering applications and computational methods used to analyze the dynamics of heart valve closures in healthy and diseased states. Computational methods are a cost-effective tool that can be used to evaluate the flow parameters of heart valves. Valve repair and replacement have long-term stability and biocompatibility issues, highlighting the need for a more robust method for resolving valvular disease. For example, while fluid-structure interaction analyses are still scarcely utilized to study aortic valves, computational fluid dynamics is used to assess the effect of different aortic valve morphologies on velocity profiles, flow patterns, helicity, wall shear stress, and oscillatory shear index in the thoracic aorta. It has been analyzed that computational flow dynamic analyses can be integrated with other methods to create a superior, more compatible method of understanding risk and compatibility.

Credible practice of modeling and simulation in healthcare: ten rules from a multidisciplinary perspective

The complexities of modern biomedicine are rapidly increasing. Thus, modeling and simulation have become increasingly important as a strategy to understand and predict the trajectory of pathophysiology, disease genesis, and disease spread in support of clinical and policy decisions. In such cases, inappropriate or ill-placed trust in the model and simulation outcomes may result in negative outcomes, and hence illustrate the need to formalize the execution and communication of modeling and simulation practices. Although verification and validation have been generally accepted as significant components of a model’s credibility, they cannot be assumed to equate to a holistic credible practice, which includes activities that can impact comprehension and in-depth examination inherent in the development and reuse of the models. For the past several years, the Committee on Credible Practice of Modeling and Simulation in Healthcare, an interdisciplinary group seeded from a U.S. interagency initiative, has worked to codify best practices. Here, we provide Ten Rules for credible practice of modeling and simulation in healthcare developed from a comparative analysis by the Committee’s multidisciplinary membership, followed by a large stakeholder community survey. These rules establish a unified conceptual framework for modeling and simulation design, implementation, evaluation, dissemination and usage across the modeling and simulation life-cycle. While biomedical science and clinical care domains have somewhat different requirements and expectations for credible practice, our study converged on rules that would be useful across a broad swath of model types. In brief, the rules are: (1) Define context clearly. (2) Use contextually appropriate data. (3) Evaluate within context. (4) List limitations explicitly. (5) Use version control. (6) Document appropriately. (7) Disseminate broadly. (8) Get independent reviews. (9) Test competing implementations. (10) Conform to standards. Although some of these are common sense guidelines, we have found that many are often missed or misconstrued, even by seasoned practitioners. Computational models are already widely used in basic science to generate new biomedical knowledge. As they penetrate clinical care and healthcare policy, contributing to personalized and precision medicine, clinical safety will require established guidelines for the credible practice of modeling and simulation in healthcare.

Deciphering the "Art" in Modeling and Simulation of the Knee Joint: Overall Strategy

Recent explorations of knee biomechanics have benefited from computational modeling, specifically leveraging advancements in finite element analysis and rigid body dynamics of joint and tissue mechanics. A large number of models have emerged with different levels of fidelity in anatomical and mechanical representation. Adapted modeling and simulation processes vary widely, based on justifiable choices in relation to anticipated use of the model. However, there are situations where modelers' decisions seem to be subjective, arbitrary, and difficult to rationalize. Regardless of the basis, these decisions form the "art" of modeling, which impact the conclusions of simulation-based studies on knee function. These decisions may also hinder the reproducibility of models and simulations, impeding their broader use in areas such as clinical decision making and personalized medicine. This document summarizes an ongoing project that aims to capture the modeling and simulation workflow in its entirety-operation procedures, deviations, models, by-products of modeling, simulation results, and comparative evaluations of case studies and applications. The ultimate goal of the project is to delineate the art of a cohort of knee modeling teams through a publicly accessible, transparent approach and begin to unravel the complex array of factors that may lead to a lack of reproducibility. This manuscript outlines our approach along with progress made so far. Potential implications on reproducibility, on science, engineering, and training of modeling and simulation, on modeling standards, and on regulatory affairs are also noted.

Perspectives on Sharing Models and Related Resources in Computational Biomechanics Research

The role of computational modeling for biomechanics research and related clinical care will be increasingly prominent. The biomechanics community has been developing computational models routinely for exploration of the mechanics and mechanobiology of diverse biological structures. As a result, a large array of models, data, and discipline-specific simulation software has emerged to support endeavors in computational biomechanics. Sharing computational models and related data and simulation software has first become a utilitarian interest, and now, it is a necessity. Exchange of models, in support of knowledge exchange provided by scholarly publishing, has important implications. Specifically, model sharing can facilitate assessment of reproducibility in computational biomechanics and can provide an opportunity for repurposing and reuse, and a venue for medical training. The community's desire to investigate biological and biomechanical phenomena crossing multiple systems, scales, and physical domains, also motivates sharing of modeling resources as blending of models developed by domain experts will be a required step for comprehensive simulation studies as well as the enhancement of their rigor and reproducibility. The goal of this paper is to understand current perspectives in the biomechanics community for the sharing of computational models and related resources. Opinions on opportunities, challenges, and pathways to model sharing, particularly as part of the scholarly publishing workflow, were sought. A group of journal editors and a handful of investigators active in computational biomechanics were approached to collect short opinion pieces as a part of a larger effort of the IEEE EMBS Computational Biology and the Physiome Technical Committee to address model reproducibility through publications. A synthesis of these opinion pieces indicates that the community recognizes the necessity and usefulness of model sharing. There is a strong will to facilitate model sharing, and there are corresponding initiatives by the scientific journals. Outside the publishing enterprise, infrastructure to facilitate model sharing in biomechanics exists, and simulation software developers are interested in accommodating the community's needs for sharing of modeling resources. Encouragement for the use of standardized markups, concerns related to quality assurance, acknowledgement of increased burden, and importance of stewardship of resources are noted. In the short-term, it is advisable that the community builds upon recent strategies and experiments with new pathways for continued demonstration of model sharing, its promotion, and its utility. Nonetheless, the need for a long-term strategy to unify approaches in sharing computational models and related resources is acknowledged. Development of a sustainable platform supported by a culture of open model sharing will likely evolve through continued and inclusive discussions bringing all stakeholders at the table, e.g., by possibly establishing a consortium.