A knowledge base of clinical trial eligibility criteria

Artificial intelligence for optimizing recruitment and retention in clinical trials: a scoping review

OBJECTIVE

The objective of our research is to conduct a comprehensive review that aims to systematically map, describe, and summarize the current utilization of artificial intelligence (AI) in the recruitment and retention of participants in clinical trials.

MATERIALS AND METHODS

A comprehensive electronic search was conducted using the search strategy developed by the authors. The search encompassed research published in English, without any time limitations, which utilizes AI in the recruitment process of clinical trials. Data extraction was performed using a data charting table, which included publication details, study design, and specific outcomes/results.

RESULTS

The search yielded 5731 articles, of which 51 were included. All the studies were designed specifically for optimizing recruitment in clinical trials and were published between 2004 and 2023. Oncology was the most covered clinical area. Applying AI to recruitment in clinical trials has demonstrated several positive outcomes, such as increasing efficiency, cost savings, improving recruitment, accuracy, patient satisfaction, and creating user-friendly interfaces. It also raises various technical and ethical issues, such as limited quantity and quality of sample size, privacy, data security, transparency, discrimination, and selection bias.

DISCUSSION AND CONCLUSION

While AI holds promise for optimizing recruitment in clinical trials, its effectiveness requires further validation. Future research should focus on using valid and standardized outcome measures, methodologically improving the rigor of the research carried out.

Optimizing Clinical Trial Eligibility Design Using Natural Language Processing Models and Real-World Data: Algorithm Development and Validation

BACKGROUND

Clinical trials are vital for developing new therapies but can also delay drug development. Efficient trial data management, optimized trial protocol, and accurate patient identification are critical for reducing trial timelines. Natural language processing (NLP) has the potential to achieve these objectives.

OBJECTIVE

This study aims to assess the feasibility of using data-driven approaches to optimize clinical trial protocol design and identify eligible patients. This involves creating a comprehensive eligibility criteria knowledge base integrated within electronic health records using deep learning-based NLP techniques.

METHODS

We obtained data of 3281 industry-sponsored phase 2 or 3 interventional clinical trials recruiting patients with non-small cell lung cancer, prostate cancer, breast cancer, multiple myeloma, ulcerative colitis, and Crohn disease from ClinicalTrials.gov, spanning the period between 2013 and 2020. A customized bidirectional long short-term memory- and conditional random field-based NLP pipeline was used to extract all eligibility criteria attributes and convert hypernym concepts into computable hyponyms along with their corresponding values. To illustrate the simulation of clinical trial design for optimization purposes, we selected a subset of patients with non-small cell lung cancer (n=2775), curated from the Mount Sinai Health System, as a pilot study.

RESULTS

We manually annotated the clinical trial eligibility corpus (485/3281, 14.78% trials) and constructed an eligibility criteria-specific ontology. Our customized NLP pipeline, developed based on the eligibility criteria-specific ontology that we created through manual annotation, achieved high precision (0.91, range 0.67-1.00) and recall (0.79, range 0.50-1) scores, as well as a high F1-score (0.83, range 0.67-1), enabling the efficient extraction of granular criteria entities and relevant attributes from 3281 clinical trials. A standardized eligibility criteria knowledge base, compatible with electronic health records, was developed by transforming hypernym concepts into machine-interpretable hyponyms along with their corresponding values. In addition, an interface prototype demonstrated the practicality of leveraging real-world data for optimizing clinical trial protocols and identifying eligible patients.

CONCLUSIONS

Our customized NLP pipeline successfully generated a standardized eligibility criteria knowledge base by transforming hypernym criteria into machine-readable hyponyms along with their corresponding values. A prototype interface integrating real-world patient information allows us to assess the impact of each eligibility criterion on the number of patients eligible for the trial. Leveraging NLP and real-world data in a data-driven approach holds promise for streamlining the overall clinical trial process, optimizing processes, and improving efficiency in patient identification.

The Role of Artificial Intelligence in Healthcare: Enhancing Coronary Computed Tomography Angiography for Coronary Artery Disease Management

This review aims to explore the potential of artificial intelligence (AI) in coronary CT angiography (CCTA), a key tool for diagnosing coronary artery disease (CAD). Because CAD is still a major cause of death worldwide, effective and accurate diagnostic methods are required to identify and manage the condition. CCTA certainly is a noninvasive alternative for diagnosing CAD, but it requires a large amount of data as input. We intend to discuss the idea of incorporating AI into CCTA, which enhances its diagnostic accuracy and operational efficiency. Using such AI technologies as machine learning (ML) and deep learning (DL) tools, CCTA images are automated to perfection and the analysis is significantly refined. It enables the characterization of a plaque, assesses the severity of the stenosis, and makes more accurate risk stratifications than traditional methods, with pinpoint accuracy. Automating routine tasks through AI-driven CCTA will reduce the radiologists' workload considerably, which is a standard benefit of such technologies. More importantly, it would enable radiologists to allocate more time and expertise to complex cases, thereby improving overall patient care. However, the field of AI in CCTA is not without its challenges, which include data protection, algorithm transparency, as well as criteria for standardization encoding. Despite such obstacles, it appears that the integration of AI technology into CCTA in the future holds great promise for keeping CAD itself in check, thereby aiding the fight against this disease and begetting better clinical outcomes and more optimized modes of healthcare. Future research on AI algorithms for CCTA, making ethical use of AI, and thereby overcoming the technical and clinical barriers to widespread adoption of this new tool, will hopefully pave the way for profound AI-driven transformations in healthcare.

recruIT: A cloud-native clinical trial recruitment support system based on Health Level 7 Fast Healthcare Interoperability Resources (HL7 FHIR) and the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM)

BACKGROUND

Clinical trials (CTs) are foundational to the advancement of evidence-based medicine and recruiting a sufficient number of participants is one of the crucial steps to their successful conduct. Yet, poor recruitment remains the most frequent reason for premature discontinuation or costly extension of clinical trials.

METHODS

We designed and implemented a novel, open-source software system to support the recruitment process in clinical trials by generating automatic recruitment recommendations. The development is guided by modern, cloud-native design principles and based on Health Level 7 (HL7) Fast Healthcare Interoperability Resources (FHIR) as an interoperability standard with the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) being used as a source of patient data. We evaluated the usability using the system usability scale (SUS) after deploying the application for use by study personnel.

RESULTS

The implementation is based on the OMOP CDM as a repository of patient data that is continuously queried for possible trial candidates based on given clinical trial eligibility criteria. A web-based screening list can be used to display the candidates and email notifications about possible new trial participants can be sent automatically. All interactions between services use HL7 FHIR as the communication standard. The system can be installed using standard container technology and supports more sophisticated deployments on Kubernetes clusters. End-users (n = 19) rated the system with a SUS score of 79.9/100.

CONCLUSION

We contribute a novel, open-source implementation to support the patient recruitment process in clinical trials that can be deployed using state-of-the art technologies. According to the SUS score, the system provides good usability.

Evaluation of Eligibility Criteria Relevance for the Purpose of IT-Supported Trial Recruitment: Descriptive Quantitative Analysis

BACKGROUND

Clinical trials (CTs) are crucial for medical research; however, they frequently fall short of the requisite number of participants who meet all eligibility criteria (EC). A clinical trial recruitment support system (CTRSS) is developed to help identify potential participants by performing a search on a specific data pool. The accuracy of the search results is directly related to the quality of the data used for comparison. Data accessibility can present challenges, making it crucial to identify the necessary data for a CTRSS to query. Prior research has examined the data elements frequently used in CT EC but has not evaluated which criteria are actually used to search for participants. Although all EC must be met to enroll a person in a CT, not all criteria have the same importance when searching for potential participants in an existing data pool, such as an electronic health record, because some of the criteria are only relevant at the time of enrollment.

OBJECTIVE

In this study, we investigated which groups of data elements are relevant in practice for finding suitable participants and whether there are typical elements that are not relevant and can therefore be omitted.

METHODS

We asked trial experts and CTRSS developers to first categorize the EC of their CTs according to data element groups and then to classify them into 1 of 3 categories: necessary, complementary, and irrelevant. In addition, the experts assessed whether a criterion was documented (on paper or digitally) or whether it was information known only to the treating physicians or patients.

RESULTS

We reviewed 82 CTs with 1132 unique EC. Of these 1132 EC, 350 (30.9%) were considered necessary, 224 (19.8%) complementary, and 341 (30.1%) total irrelevant. To identify the most relevant data elements, we introduced the data element relevance index (DERI). This describes the percentage of studies in which the corresponding data element occurs and is also classified as necessary or supplementary. We found that the query of "diagnosis" was relevant for finding participants in 79 (96.3%) of the CTs. This group was followed by "date of birth/age" with a DERI of 85.4% (n=70) and "procedure" with a DERI of 35.4% (n=29).

CONCLUSIONS

The distribution of data element groups in CTs has been heterogeneously described in previous works. Therefore, we recommend identifying the percentage of CTs in which data element groups can be found as a more reliable way to determine the relevance of EC. Only necessary and complementary criteria should be included in this DERI.

Text Classification of Cancer Clinical Trial Eligibility Criteria

Automatic identification of clinical trials for which a patient is eligible is complicated by the fact that trial eligibility are stated in natural language. A potential solution to this problem is to employ text classification methods for common types of eligibility criteria. In this study, we focus on seven common exclusion criteria in cancer trials: prior malignancy, human immunodeficiency virus, hepatitis B, hepatitis C, psychiatric illness, drug/substance abuse, and autoimmune illness. Our dataset consists of 764 phase III cancer trials with these exclusions annotated at the trial level. We experiment with common transformer models as well as a new pre-trained clinical trial BERT model. Our results demonstrate the feasibility of automatically classifying common exclusion criteria. Additionally, we demonstrate the value of a pre-trained language model specifically for clinical trials, which yield the highest average performance across all criteria.

A review of research on eligibility criteria for clinical trials

The purpose of this paper is to systematically sort out and analyze the cutting-edge research on the eligibility criteria of clinical trials. Eligibility criteria are important prerequisites for the success of clinical trials. It directly affects the final results of the clinical trials. Inappropriate eligibility criteria will lead to insufficient recruitment, which is an important reason for the eventual failure of many clinical trials. We have investigated the research status of eligibility criteria for clinical trials on academic platforms such as arXiv and NIH. We have classified and sorted out all the papers we found, so that readers can understand the frontier research in this field. Eligibility criteria are the most important part of a clinical trial study. The ultimate goal of research in this field is to formulate more scientific and reasonable eligibility criteria and speed up the clinical trial process. The global research on the eligibility criteria of clinical trials is mainly divided into four main aspects: natural language processing, patient pre-screening, standard evaluation, and clinical trial query. Compared with the past, people are now using new technologies to study eligibility criteria from a new perspective (big data). In the research process, complex disease concepts, how to choose a suitable dataset, how to prove the validity and scientific of the research results, are challenges faced by researchers (especially for computer-related researchers). Future research will focus on the selection and improvement of artificial intelligence algorithms related to clinical trials and related practical applications such as databases, knowledge graphs, and dictionaries.

A data-driven approach to optimizing clinical study eligibility criteria

OBJECTIVE

Feasible, safe, and inclusive eligibility criteria are crucial to successful clinical research recruitment. Existing expert-centered methods for eligibility criteria selection may not be representative of real-world populations. This paper presents a novel model called OPTEC (OPTimal Eligibility Criteria) based on the Multiple Attribute Decision Making method boosted by an efficient greedy algorithm.

METHODS

It systematically identifies the optimal criteria combination for a given medical condition with the optimal tradeoff among feasibility, patient safety, and cohort diversity. The model offers flexibility in attribute configurations and generalizability to various clinical domains. The model was evaluated on two clinical domains (i.e., Alzheimer's disease and Neoplasm of pancreas) using two datasets (i.e., MIMIC-III dataset and NewYork-Presbyterian/Columbia University Irving Medical Center (NYP/CUIMC) database).

RESULTS

We simulated the process of automatically optimizing eligibility criteria according to user-specified prioritization preferences and generated recommendations based on the top-ranked criteria combination accordingly (top 0.41-2.75%) with OPTEC. Harnessing the power of the model, we designed an interactive criteria recommendation system and conducted a case study with an experienced clinical researcher using the think-aloud protocol.

CONCLUSIONS

The results demonstrated that OPTEC could be used to recommend feasible eligibility criteria combinations, and to provide actionable recommendations for clinical study designers to construct a feasible, safe, and diverse cohort definition during early study design.

How to Improve Cancer Patients ENrollment in Clinical Trials From rEal-Life Databases Using the Observational Medical Outcomes Partnership Oncology Extension: Results of the PENELOPE Initiative in Urologic Cancers

PURPOSE

To compare the computability of Observational Medical Outcomes Partnership (OMOP)-based queries related to prescreening of patients using two versions of the OMOP common data model (CDM; v5.3 and v5.4) and to assess the performance of the Greater Paris University Hospital (APHP) prescreening tool.

MATERIALS AND METHODS

We identified the prescreening information items being relevant for prescreening of patients with cancer. We randomly selected 15 academic and industry-sponsored urology phase I-IV clinical trials (CTs) launched at APHP between 2016 and 2021. The computability of the related prescreening criteria (PC) was defined by their translation rate in OMOP-compliant queries and by their execution rate on the APHP clinical data warehouse (CDW) containing data of 205,977 patients with cancer. The overall performance of the prescreening tool was assessed by the rate of true- and false-positive cases of three randomly selected CTs.

RESULTS

We defined a list of 15 minimal information items being relevant for patients' prescreening. We identified 83 PC of the 534 eligibility criteria from the 15 CTs. We translated 33 and 62 PC in queries on the basis of OMOP CDM v5.3 and v5.4, respectively (translation rates of 40% and 75%, respectively). Of the 33 PC translated in the v5.3 of the OMOP CDM, 19 could be executed on the APHP CDW (execution rate of 58%). Of 83 PC, the computability rate on the APHP CDW reached 23%. On the basis of three CTs, we identified 17, 32, and 63 patients as being potentially eligible for inclusion in those CTs, resulting in positive predictive values of 53%, 41%, and 21%, respectively.

CONCLUSION

We showed that PC could be formalized according to the OMOP CDM and that the oncology extension increased their translation rate through better representation of cancer natural history.

Clinical research staff perceptions on a natural language processing-driven tool for eligibility prescreening: An iterative usability assessment

BACKGROUND

Participant recruitment is a barrier to successful clinical research. One strategy to improve recruitment is to conduct eligibility prescreening, a resource-intensive process where clinical research staff manually reviews electronic health records data to identify potentially eligible patients. Criteria2Query (C2Q) was developed to address this problem by capitalizing on natural language processing to generate queries to identify eligible participants from clinical databases semi-autonomously.

OBJECTIVE

We examined the clinical research staff's perceived usability of C2Q for clinical research eligibility prescreening.

METHODS

Twenty clinical research staff evaluated the usability of C2Q using a cognitive walkthrough with a think-aloud protocol and a Post-Study System Usability Questionnaire. On-screen activity and audio were recorded and transcribed. After every-five evaluators completed an evaluation, usability problems were rated by informatics experts and prioritized for system refinement. There were four iterations of system refinement based on the evaluation feedback. Guided by the Organizational Framework for Intuitive Human-computer Interaction, we performed a directed deductive content analysis of the verbatim transcriptions.

RESULTS

Evaluators aged from 24 to 46 years old (33.8; SD: 7.32) demonstrated high computer literacy (6.36; SD:0.17); female (75 %), White (35 %), and clinical research coordinators (45 %). C2Q demonstrated high usability during the final cycle (2.26 out of 7 [lower scores are better], SD: 0.74). The number of unique usability issues decreased after each refinement. Fourteen subthemes emerged from three themes: seeking user goals, performing well-learned tasks, and determining what to do next.

CONCLUSIONS

The cognitive walkthrough with a think-aloud protocol informed iterative system refinement and demonstrated the usability of C2Q by clinical research staff. Key recommendations for system development and implementation include improving system intuitiveness and overall user experience through comprehensive consideration of user needs and requirements for task completion.

Artificial Intelligence Applied to clinical trials: opportunities and challenges

Background

Clinical Trials (CTs) remain the foundation of safe and effective drug development. Given the evolving data-driven and personalized medicine approach in healthcare, it is imperative for companies and regulators to utilize tailored Artificial Intelligence (AI) solutions that enable expeditious and streamlined clinical research. In this paper, we identified opportunities, challenges, and potential implications of AI in CTs.

Methods

Following an extensive search in relevant databases and websites, we gathered publications tackling the use of AI and Machine Learning (ML) in CTs from the past 5 years in the US and Europe, including Regulatory Authorities' documents.

Results

Documented applications of AI commonly concern the oncology field and are mostly being applied in the area of recruitment. Main opportunities discussed aim to create efficiencies across CT activities, including the ability to reduce sample sizes, improve enrollment and conduct faster, more optimized adaptive CTs. While AI is an area of enthusiastic development, the identified challenges are ethical in nature and relate to data availability, standards, and most importantly, lack of regulatory guidance hindering the acceptance of AI tools in drug development. However, future implications are significant and are anticipated to improve the probability of success, reduce trial burden and overall, speed up research and regulatory approval.

Conclusion

The use of AI in CTs is in its relative infancy; however, it is a fast-evolving field. As regulators provide more guidance on the acceptability of AI in specific areas, we anticipate the scope of use to broaden and the volume of implementation to increase rapidly.

Clinical Research Informatics

OBJECTIVES

To summarize key contributions to current research in the field of Clinical Research Informatics (CRI) and to select best papers published in 2021.

METHOD

Using PubMed, we did a bibliographic search using a combination of MeSH descriptors and free-text terms on CRI, followed by a double-blind review in order to select a list of candidate best papers to be peer-reviewed by external reviewers. After peer-review ranking, three section editors met for a consensus meeting and the editorial team was organized to finally conclude on the selected three best papers.

RESULTS

Among the 1,096 papers (published in 2021) returned by the search and in the scope of the various areas of CRI, the full review process selected three best papers. The first best paper describes an operational and scalable framework for generating EHR datasets based on a detailed clinical model with an application in the domain of the COVID-19 pandemics. The authors of the second best paper present a secure and scalable platform for the preprocessing of biomedical data for deep data-driven health management applied for the detection of pre-symptomatic COVID-19 cases and for biological characterization of insulin-resistance heterogeneity. The third best paper provides a contribution to the integration of care and research activities with the REDCap Clinical Data and Interoperability sServices (CDIS) module improving the accuracy and efficiency of data collection.

CONCLUSIONS

The COVID-19 pandemic is still significantly stimulating research efforts in the CRI field to improve the process deeply and widely for conducting real-world studies as well as for optimizing clinical trials, the duration and cost of which are constantly increasing. The current health crisis highlights the need for healthcare institutions to continue the development and deployment of Big Data spaces, to strengthen their expertise in data science and to implement efficient data quality evaluation and improvement programs.

Year 2021: COVID-19, Information Extraction and BERTization among the Hottest Topics in Medical Natural Language Processing

OBJECTIVES

Analyze the content of publications within the medical natural language processing (NLP) domain in 2021.

METHODS

Automatic and manual preselection of publications to be reviewed, and selection of the best NLP papers of the year. Analysis of the important issues.

RESULTS

Four best papers have been selected in 2021. We also propose an analysis of the content of the NLP publications in 2021, all topics included.

CONCLUSIONS

The main issues addressed in 2021 are related to the investigation of COVID-related questions and to the further adaptation and use of transformer models. Besides, the trends from the past years continue, such as information extraction and use of information from social networks.

Machine Learning Prediction of Clinical Trial Operational Efficiency

Clinical trials are the gatekeepers and bottlenecks of progress in medicine. In recent years, they have become increasingly complex and expensive, driven by a growing number of stakeholders requiring more endpoints, more diverse patient populations, and a stringent regulatory environment. Trial designers have historically relied on investigator expertise and legacy norms established within sponsor companies to improve operational efficiency while achieving study goals. As such, data-driven forecasts of operational metrics can be a useful resource for trial design and planning. We develop a machine learning model to predict clinical trial operational efficiency using a novel dataset from Roche containing over 2,000 clinical trials across 20 years and multiple disease areas. The data includes important operational metrics related to patient recruitment and trial duration, as well as a variety of trial features such as the number of procedures, eligibility criteria, and endpoints. Our results demonstrate that operational efficiency can be predicted robustly using trial features, which can provide useful insights to trial designers on the potential impact of their decisions on patient recruitment success and trial duration.

How can natural language processing help model informed drug development?: a review

Objective

To summarize applications of natural language processing (NLP) in model informed drug development (MIDD) and identify potential areas of improvement.

Materials and Methods

Publications found on PubMed and Google Scholar, websites and GitHub repositories for NLP libraries and models. Publications describing applications of NLP in MIDD were reviewed. The applications were stratified into 3 stages: drug discovery, clinical trials, and pharmacovigilance. Key NLP functionalities used for these applications were assessed. Programming libraries and open-source resources for the implementation of NLP functionalities in MIDD were identified.

Results

NLP has been utilized to aid various processes in drug development lifecycle such as gene-disease mapping, biomarker discovery, patient-trial matching, adverse drug events detection, etc. These applications commonly use NLP functionalities of named entity recognition, word embeddings, entity resolution, assertion status detection, relation extraction, and topic modeling. The current state-of-the-art for implementing these functionalities in MIDD applications are transformer models that utilize transfer learning for enhanced performance. Various libraries in python, R, and Java like huggingface, sparkNLP, and KoRpus as well as open-source platforms such as DisGeNet, DeepEnroll, and Transmol have enabled convenient implementation of NLP models to MIDD applications.

Discussion

Challenges such as reproducibility, explainability, fairness, limited data, limited language-support, and security need to be overcome to ensure wider adoption of NLP in MIDD landscape. There are opportunities to improve the performance of existing models and expand the use of NLP in newer areas of MIDD.

Conclusions

This review provides an overview of the potential and pitfalls of current NLP approaches in MIDD.

A knowledge graph of clinical trials ([Formula: see text])

Effective and successful clinical trials are essential in developing new drugs and advancing new treatments. However, clinical trials are very expensive and easy to fail. The high cost and low success rate of clinical trials motivate research on inferring knowledge from existing clinical trials in innovative ways for designing future clinical trials. In this manuscript, we present our efforts on constructing the first publicly available Clinical Trials Knowledge Graph, denoted as [Formula: see text]. [Formula: see text] includes nodes representing medical entities in clinical trials (e.g., studies, drugs and conditions), and edges representing the relations among these entities (e.g., drugs used in studies). Our embedding analysis demonstrates the potential utilities of [Formula: see text] in various applications such as drug repurposing and similarity search, among others.

Leveraging electronic health record data for clinical trial planning by assessing eligibility criteria's impact on patient count and safety

OBJECTIVE

To present an approach on using electronic health record (EHR) data that assesses how different eligibility criteria, either individually or in combination, can impact patient count and safety (exemplified by all-cause hospitalization risk) and further assist with criteria selection for prospective clinical trials.

MATERIALS AND METHODS

Trials in three disease domains - relapsed/refractory (r/r) lymphoma/leukemia; hepatitis C virus (HCV); stages 3 and 4 chronic kidney disease (CKD) - were analyzed as case studies for this approach. For each disease domain, criteria were identified and all criteria combinations were used to create EHR cohorts. Per combination, two values were derived: (1) number of eligible patients meeting the selected criteria; (2) hospitalization risk, measured as the hazard ratio between those that qualified and those that did not. From these values, k-means clustering was applied to derive which criteria combinations maximized patient counts but minimized hospitalization risk.

RESULTS

Criteria combinations that reduced hospitalization risk without substantial reductions on patient counts were as follows: for r/r lymphoma/leukemia (23 trials; 9 criteria; 623 patients), applying no infection and adequate absolute neutrophil count while forgoing no prior malignancy; for HCV (15; 7; 751), applying no human immunodeficiency virus and no hepatocellular carcinoma while forgoing no decompensated liver disease/cirrhosis; for CKD (10; 9; 23893), applying no congestive heart failure.

CONCLUSIONS

Within each disease domain, the more drastic effects were generally driven by a few criteria. Similar criteria across different disease domains introduce different changes. Although results are contingent on the trial sample and the EHR data used, this approach demonstrates how EHR data can inform the impact on safety and available patients when exploring different criteria combinations for designing clinical trials.

Prediction of clinical trial enrollment rates

Clinical trials represent a critical milestone of translational and clinical sciences. However, poor recruitment to clinical trials has been a long standing problem affecting institutions all over the world. One way to reduce the cost incurred by insufficient enrollment is to minimize initiating trials that are most likely to fall short of their enrollment goal. Hence, the ability to predict which proposed trials will meet enrollment goals prior to the start of the trial is highly beneficial. In the current study, we leveraged a data set extracted from ClinicalTrials.gov that consists of 46,724 U.S. based clinical trials from 1990 to 2020. We constructed 4,636 candidate predictors based on data collected by ClinicalTrials.gov and external sources for enrollment rate prediction using various state-of-the-art machine learning methods. Taking advantage of a nested time series cross-validation design, our models resulted in good predictive performance that is generalizable to future data and stable over time. Moreover, information content analysis revealed the study design related features to be the most informative feature type regarding enrollment. Compared to the performance of models built with all features, the performance of models built with study design related features is only marginally worse (AUC = 0.78 ± 0.03 vs. AUC = 0.76 ± 0.02). The results presented can form the basis for data-driven decision support systems to assess whether proposed clinical trials would likely meet their enrollment goal.