The concept of risk in comparative effectiveness research

Recommendations for Reporting Machine Learning Analyses in Clinical Research

Use of machine learning (ML) in clinical research is growing steadily given the increasing availability of complex clinical data sets. ML presents important advantages in terms of predictive performance and identifying undiscovered subpopulations of patients with specific physiology and prognoses. Despite this popularity, many clinicians and researchers are not yet familiar with evaluating and interpreting ML analyses. Consequently, readers and peer-reviewers alike may either overestimate or underestimate the validity and credibility of an ML-based model. Conversely, ML experts without clinical experience may present details of the analysis that are too granular for a clinical readership to assess. Overwhelming evidence has shown poor reproducibility and reporting of ML models in clinical research suggesting the need for ML analyses to be presented in a clear, concise, and comprehensible manner to facilitate understanding and critical evaluation. We present a recommendation for transparent and structured reporting of ML analysis results specifically directed at clinical researchers. Furthermore, we provide a list of key reporting elements with examples that can be used as a template when preparing and submitting ML-based manuscripts for the same audience.

Harmonization and streamlining of research oversight for pragmatic clinical trials

The oversight of research involving human participants is a complex process that requires institutional review board review as well as multiple non-institutional review board institutional reviews. This multifaceted process is particularly challenging for multisite research when each site independently completes all required local reviews. The lack of inter-institutional standardization can result in different review outcomes for the same protocol, which can delay study operations from start-up to study completion. Hence, there have been strong calls to harmonize and thus streamline the research oversight process. Although the institutional review board is only one of the required reviews, it is often identified as the target for harmonization and streamlining. Data regarding variability in decision-making and interpretation of the regulations across institutional review boards have led to a perception that variability among institutional review boards is a primary contributor to the problems with review of multisite research. In response, many researchers and policymakers have proposed the use of a single institutional review board of record, also called a central institutional review board, as an important remedy. While this proposal has merit, the use of a central institutional review board for multisite research does not address the larger problem of completing non-institutional review board institutional review in addition to institutional review board review—and coordinating the interdependence of these reviews. In this article, we describe the overall research oversight process, distinguish between institutional review board and institutional responsibilities, and identify challenges and opportunities for harmonization and streamlining. We focus on procedural and organizational issues and presume that the protection of human subjects remains the paramount concern. Suggested modifications of institutional review board processes that focus on time, efficiency, and consistency of review must also address what effect such changes have on the quality of review. We acknowledge that assessment of quality is difficult in that quality metrics for institutional review board review remain elusive. At best, we may be able to assess the time it takes to review protocols and the consistency across institutions.

The Food and Drug Administration and pragmatic clinical trials of marketed medical products

Pragmatic clinical trials can help answer questions of comparative effectiveness for interventions routinely used in medical practice. Pragmatic clinical trials may examine outcomes of one or more marketed medical products, and they are heterogeneous in design and risk. The Food and Drug Administration is charged with protecting the rights, safety, and welfare of individuals enrolled in clinical investigations, as well as assuring the integrity of the data upon which approval of medical products is made. The Food and Drug Administration has broad jurisdiction over drugs and medical devices (whether or not they are approved for marketing), and as such, clinical investigations of these products are subject to applicable Food and Drug Administration regulations. While many pragmatic clinical trials will meet the criteria for an exemption from the requirements for an investigational new drug application or investigational device exemption, in general, all clinical investigations of medical products that fall under Food and Drug Administration jurisdiction must adhere to regulations for informed consent and review by an institutional review board. We are concerned that current Food and Drug Administration requirements for obtaining individual informed consent may deter or delay the conduct of pragmatic clinical trials intended to develop reliable evidence of comparative safety and effectiveness of approved medical products that are regulated by the Food and Drug Administration. Under current regulations, there are no described mechanisms to alter or waive informed consent to make it less burdensome or more practicable for low-risk pragmatic clinical trials. We recommend that the Food and Drug Administration establish a risk-based approach to obtaining informed consent in pragmatic clinical trials that would facilitate the conduct of pragmatic clinical trials without compromising the protection of enrolled individuals or the integrity of the resulting data.