A systematic review of the processes used to link clinical trial registrations to their published results

Evidence-b(i)ased practice: Selective and inadequate reporting in early childhood autism intervention research

LAY ABSTRACT

When researchers fail to report their findings or only report some of their findings, it can make it difficult for clinicians to provide effective intervention recommendations. However, no one has examined whether this is a problem in studies of early childhood autism interventions. We studied how researchers that study early childhood autism interventions report their findings. We found that most researchers did not register their studies when they were supposed to (before the start of the study), and that many researchers did not provide all of the needed information in the registration. We also found that researchers frequently did not publish their findings when their studies were complete. When we looked at published reports, we found that many of the studies did not report enough information, and that many studies were reported differently from their registrations, suggesting that researchers were selectively reporting positive outcomes and ignoring or misrepresenting less positive outcomes. Because we found so much evidence that researchers are failing to report their findings quickly and correctly, we suggested some practical changes to make it better.

Evaluating the discoverability of supporting research materials in ClinicalTrials.gov for US federally funded COVID-19 clinical studies

Objective

The objective of this study was to evaluate the discoverability of supporting research materials, including supporting documents, individual participant data (IPD), and associated publications, in US federally funded COVID-19 clinical study records in ClinicalTrials.gov (CTG).

Methods

Study registration records were evaluated for (1) links to supporting documents, including protocols, informed consent forms, and statistical analysis plans; (2) information on how unaffiliated researchers may access IPD and, when applicable, the linking of the IPD record back to the CTG record; and (3) links to associated publications and, when applicable, the linking of the publication record back to the CTG record.

Results

206 CTG study records were included in the analysis. Few records shared supporting documents, with only 4% of records sharing all 3 document types. 27% of records indicated they intended to share IPD, with 45% of these providing sufficient information to request access to the IPD. Only 1 dataset record was located, which linked back to its corresponding CTG record. The majority of CTG records did not have links to publications (61%), and only 21% linked out to at least 1 results publication. All publication records linked back to their corresponding CTG records.

Conclusion

With only 4% of records sharing all supporting document types, 12% sufficient information to access IPD, and 21% results publications, improvements can be made to the discoverability of research materials in federally funded, COVID-19 CTG records. Sharing these materials on CTG can increase their discoverability, therefore increasing the validity, transparency, and reusability of clinical research.

Linking Cancer Clinical Trials to their Result Publications

The results of clinical trials are a valuable source of evidence for researchers, policy makers, and healthcare professionals. However, online trial registries do not always contain links to the publications that report on their results, instead requiring a time-consuming manual search. Here, we explored the application of pre-trained transformer-based language models to automatically identify result-reporting publications of cancer clinical trials by computing dense vectors and performing semantic search. Models were fine-tuned on text data from trial registry fields and article metadata using a contrastive learning approach. The best performing model was PubMedBERT, which achieved a mean average precision of 0.592 and ranked 70.3% of a trial's publications in the top 5 results when tested on the holdout test trials. Our results suggest that semantic search using embeddings from transformer models may be an effective approach to the task of linking trials to their publications.

Availability of results of clinical trials registered on EU Clinical Trials Register: cross sectional audit study

Objective

To identify the availability of results for trials registered on the European Union Clinical Trials Register (EUCTR) compared with other dissemination routes to understand its value as a results repository.

Design

Cross sectional audit study.

Setting

EUCTR protocols and results sections, data extracted 1-3 December 2020.

Population

Random sample of 500 trials registered on EUCTR with a completion date of more than two years from the beginning of searches (ie, 1 December 2018).

Main outcome measures

Proportion of trials with results across the examined dissemination routes (EUCTR, ClinicalTrials.gov, ISRCTN registry, and journal publications), and for each dissemination route individually. Prespecified secondary outcomes were number and proportion of unique results, and the timing of results, for each dissemination route.

Results

In the sample of 500 trials, availability of results on EUCTR (53.2%, 95% confidence interval 48.8% to 57.6%) was similar to the peer reviewed literature (58.6%, 54.3% to 62.9%) and exceeded the proportion of results available on other registries with matched records. Among the 383 trials with any results, 55 (14.4%, 10.9% to 17.9%) were only available on EUCTR. Also, after the launch of the EUCTR results database, median time to results was fastest on EUCTR (1142 days, 95% confidence interval 812 to 1492), comparable with journal publications (1226 days, 1074 to 1551), and exceeding ClinicalTrials.gov (3321 days, 1653 to undefined). For 117 trials (23.4%, 19.7% to 27.1%), however, results were published elsewhere but not submitted to the EUCTR registry, and no results were located in any dissemination route for 117 trials (23.4%, 19.7% to 27.1).

Conclusions

EUCTR should be considered in results searches for systematic reviews and can help researchers and the public to access the results of clinical trials, unavailable elsewhere, in a timely way. Reporting requirements, such as the EU's, can help in avoiding research waste by ensuring results are reported. The registry's true value, however, is unrealised because of inadequate compliance with EU guidelines, and problems with data quality that complicate the routine use of the registry. As the EU transitions to a new registry, continuing to emphasise the importance of EUCTR and the provision of timely and complete data is critical. For the future, EUCTR will still hold important information from the past two decades of clinical research in Europe. With increased efforts from sponsors and regulators, the registry can continue to grow as a source of results of clinical trials, many of which might be unavailable from other dissemination routes.

A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses

Living reviews are an increasingly popular research paradigm. The purpose of a 'living' approach is to allow rapid collation, appraisal and synthesis of evolving evidence on an important research topic, enabling timely influence on patient care and public health policy. However, living reviews are time- and resource-intensive. The accumulation of new evidence and the possibility of developments within the review's research topic can introduce unique challenges into the living review workflow. To investigate the potential of software tools to support living systematic or rapid reviews, we present a narrative review informed by an examination of tools contained on the Systematic Review Toolbox website. We identified 11 tools with relevant functionalities and discuss the important features of these tools with respect to different steps of the living review workflow. Four tools (NestedKnowledge, SWIFT-ActiveScreener, DistillerSR, EPPI-Reviewer) covered multiple, successive steps of the review process, and the remaining tools addressed specific components of the workflow, including scoping and protocol formulation, reference retrieval, automated data extraction, write-up and dissemination of data. We identify several ways in which living reviews can be made more efficient and practical. Most of these focus on general workflow management, or automation through artificial intelligence and machine-learning, in the screening process. More sophisticated uses of automation mostly target living rapid reviews to increase the speed of production or evidence maps to broaden the scope of the map. We use a case study to highlight some of the barriers and challenges to incorporating tools into the living review workflow and processes. These include increased workload, the need for organisation, ensuring timely dissemination and challenges related to the development of bespoke automation tools to facilitate the review process. We describe how current end-user tools address these challenges, and which knowledge gaps remain that could be addressed by future tool development. Dedicated web presences for automatic dissemination of in-progress evidence updates, rather than solely relying on peer-reviewed journal publications, help to make the effort of a living evidence synthesis worthwhile. Despite offering basic living review functionalities, existing end-user tools could be further developed to be interoperable with other tools to support multiple workflow steps seamlessly, to address broader automatic evidence retrieval from a larger variety of sources, and to improve dissemination of evidence between review updates.

Time to Publication for Randomized Clinical Trials Presented as Abstracts at Three Gastroenterology and Hepatology Conferences in 2017

Background & Aims

Results of randomized clinical trials are often first presented as conference abstracts, but these abstracts may be difficult to find, and trial results included in the abstract may not be followed by subsequent journal publications. In a review of abstracts submitted to eight major medical and surgical conferences in 2017, we identified 237 abstracts reporting primary results of randomized clinical trials accepted for presentation at three major gastroenterology and hepatology conferences. The aims of this new analysis were to determine the publication rate for these abstracts and the proportion of publications that included trial registration numbers in the publication abstract.

Methods

Clinical trial registries, PubMed, Europe PMC, and Google Scholar were searched through November 1, 2021, for publications reporting trial results for the selected abstracts. Publications were reviewed to determine if they included a trial registration number and if the registration number was in the abstract.

Results

Publications were found for 157 abstracts (66%) within four years of the conference. Publications were found more frequently for the 194 abstracts reporting results of registered trials (144, 74%) than for the 43 abstracts reporting unregistered trials (13, 30%), but only 67% of these 144 publications included the registration number in the publication abstract. Ten unpublished trials had summary results posted on ClinicalTrials.gov.

Conclusions

Clinical trial results could be more accessible if all trials were registered, authors included registration numbers in both conference and journal abstracts, and journal editors required the inclusion of registration numbers in publication abstracts for registered clinical trials.

Predicting Publication of Clinical Trials Using Structured and Unstructured Data: Model Development and Validation Study

BACKGROUND

Publication of registered clinical trials is a critical step in the timely dissemination of trial findings. However, a significant proportion of completed clinical trials are never published, motivating the need to analyze the factors behind success or failure to publish. This could inform study design, help regulatory decision-making, and improve resource allocation. It could also enhance our understanding of bias in the publication of trials and publication trends based on the research direction or strength of the findings. Although the publication of clinical trials has been addressed in several descriptive studies at an aggregate level, there is a lack of research on the predictive analysis of a trial's publishability given an individual (planned) clinical trial description.

OBJECTIVE

We aimed to conduct a study that combined structured and unstructured features relevant to publication status in a single predictive approach. Established natural language processing techniques as well as recent pretrained language models enabled us to incorporate information from the textual descriptions of clinical trials into a machine learning approach. We were particularly interested in whether and which textual features could improve the classification accuracy for publication outcomes.

METHODS

In this study, we used metadata from ClinicalTrials.gov (a registry of clinical trials) and MEDLINE (a database of academic journal articles) to build a data set of clinical trials (N=76,950) that contained the description of a registered trial and its publication outcome (27,702/76,950, 36% published and 49,248/76,950, 64% unpublished). This is the largest data set of its kind, which we released as part of this work. The publication outcome in the data set was identified from MEDLINE based on clinical trial identifiers. We carried out a descriptive analysis and predicted the publication outcome using 2 approaches: a neural network with a large domain-specific language model and a random forest classifier using a weighted bag-of-words representation of text.

RESULTS

First, our analysis of the newly created data set corroborates several findings from the existing literature regarding attributes associated with a higher publication rate. Second, a crucial observation from our predictive modeling was that the addition of textual features (eg, eligibility criteria) offers consistent improvements over using only structured data (F₁-score=0.62-0.64 vs F₁-score=0.61 without textual features). Both pretrained language models and more basic word-based representations provide high-utility text representations, with no significant empirical difference between the two.

CONCLUSIONS

Different factors affect the publication of a registered clinical trial. Our approach to predictive modeling combines heterogeneous features, both structured and unstructured. We show that methods from natural language processing can provide effective textual features to enable more accurate prediction of publication success, which has not been explored for this task previously.

regCOVID: Tracking publications of registered COVID-19 studies

Background

In response to the COVID-19 pandemic many clinical studies have been initiated leading to the need for efficient ways to track and analyze study results. We expanded our previous project that tracked registered COVID-19 clinical studies to also track result articles generated from these studies. Our objective was to develop a data science approach to identify and analyze all publications linked to COVID-19 clinical studies and generate a prioritized list of publications for efficient understanding of the state of COVID-19 clinical research.

Methods

We conducted searches of ClinicalTrials.gov and PubMed to identify articles linked to COVID-19 studies, and developed criteria based on the trial phase, intervention, location, and record recency to develop a prioritized list of result publications.

Results

The performed searchers resulted in 1 022 articles linked to 565 interventional trials (17.8% of all 3 167 COVID-19 interventional trials as of 31 January 2022). 609 publications were identified via abstract-link in PubMed and 413 via registry-link in ClinicalTrials.gov, with 27 articles linked from both sources. Of the 565 trials publishing at least one article, 197 (34.9%) had multiple linked publications. An attention score was assigned to each publication to develop a prioritized list of all publications linked to COVID-19 trials and 83 publications were identified that are result articles from late phase (Phase 3) trials with at least one US site and multiple study record updates. For COVID-19 vaccine trials, 108 linked result articles for 64 trials (14.7% of 436 total COVID-19 vaccine trials) were found.

Conclusions

Our method allows for the efficient identification of important COVID-19 articles that report results of registered clinical trials and are connected via a structured article-trial link. Our data science methodology also allows for consistent and as needed data updates and is generalizable to other conditions of interest.

A comparison of registered and published primary outcomes in clinical trials of opioid use disorder: ACTTION review and recommendations

BACKGROUND AND AIMS

Prospective trial registration can increase research integrity. This Analgesic, Anesthetic, and Addiction Clinical Trial Translations, Innovations, Opportunities, and Networks (ACTTION) review was designed to compare the primary outcomes (PO) reported in registries with associated publications for opioid use disorder (OUD) clinical trials.

DESIGN

The World Health Organization's International Clinical Trials Registry Platform (ICTRP) was searched for completed trials (2010 through 2019). Associated publications were identified and paired with trial registry data based on the publication date.

MEASUREMENTS

Reviewers independently rated the occurrence of discrepancies between the POs in the registry compared to the publication. An analysis of prospective versus retrospective registration was also completed.

FINDINGS

One-hundred and forty trials were identified in the search, and 43 registry-publication pairs evaluated. Only 34 of the 43 pairs could be examined for discrepancies because nine did not report a PO in registry and publication. Of the 34 pairs, only four met rigorous criteria for prospective trial registration and had an exact match of POs. In contrast, the majority of the 34 trials, or 80%, had inconsistent POs (e.g., registered secondary outcomes published as primary; the timing of PO not specified) and/or were retrospectively registered.

CONCLUSIONS

Many clinical trials focused on OUD have not met the standards of trial registration, such as consistent reporting of POs and prospective registration. Failure to properly register trial characteristics undermines the validity of research findings and can delay the development of life-saving treatments. Recommendations for improving prospective trial reporting practices are provided.

Results publications are inadequately linked to trial registrations: An automated pipeline and evaluation of German university medical centers

BACKGROUND/AIMS

Informed clinical guidance and health policy relies on clinicians, policymakers, and guideline developers finding comprehensive clinical evidence and linking registrations and publications of the same clinical trial. To support the finding and linking of trial evidence, the World Health Organization, the International Committee of Medical Journal Editors, and the Consolidated Standards of Reporting Trials ask researchers to provide the trial registration number in their publication and a reference to the publication in the registration. This practice costs researchers minimal effort and makes evidence synthesis more thorough and efficient. Nevertheless, trial evidence appears inadequately linked, and the extent of trial links in Germany remains unquantified. This cross-sectional study aims to evaluate links between registrations and publications across clinical trials conducted by German university medical centers and registered in ClinicalTrials.gov or the German Clinical Trials Registry. Secondary aims are to develop an automated pipeline that can be applied to other cohorts of trial registrations and publications, and to provide stakeholders, from trialists to registries, with guidance to improve trial links.

METHODS

We used automated strategies to download and extract data from trial registries, PubMed, and results publications for a cohort of registered, published trials conducted across German university medical centers and completed between 2009 and 2017. We implemented regular expressions to detect and classify publication identifiers in registrations, and trial registration numbers in publication metadata, abstracts, and full-texts.

RESULTS

In breach of long-standing guidelines, 75% (1,418) of trials failed to reference trial registration numbers in both the abstract and full-text of the journal article in which the results were published. Furthermore, 50% (946) of trial registrations did not contain links to their results publications. Seventeen percent (327) of trials had no links, so that associating registration and publication required manual searching and screening. Overall, trials in ClinicalTrials.gov were better linked than those in the German Clinical Trials Registry; PubMed and registry infrastructures appear to drive this difference. Trial registration numbers were more likely to be transferred to PubMed metadata from abstracts for ClinicalTrials.gov trials than for German Clinical Trials Registry trials. Most (78%, 662/849) ClinicalTrials.gov registrations with a publication link were automatically indexed from PubMed metadata, which is not possible in the German Clinical Trials Registry.

CONCLUSIONS

German university medical centers have not comprehensively linked trial registrations and publications, despite established recommendations. This shortcoming threatens the quality of evidence synthesis and medical practice, and burdens researchers with manually searching and linking trial data. Researchers could easily improve this by copy-and-pasting references between their trial registrations and publications. Other stakeholders could build on this practice, for example, PubMed could capture additional trial registration numbers using automated strategies (like those developed in this study), and the German Clinical Trials Registry could automatically index publications from PubMed.

Identifying unreported links between ClinicalTrials.gov trial registrations and their published results

A substantial proportion of trial registrations are not linked to corresponding published articles, limiting analyses and new tools. Our aim was to develop a method for finding articles reporting the results of trials that are registered on ClinicalTrials.gov when they do not include metadata links. We used a set of 27,280 trial registration and article pairs to train and evaluate methods for identifying missing links in both directions-from articles to registrations and from registrations to articles. We trained a classifier with six distance metrics as feature representations to rank the correct article or registration, using recall@K to evaluate performance and compare to baseline methods. When identifying links from registrations to published articles, the classifier ranked the correct article first (recall@1) among 378,048 articles in 80.8% of evaluation cases and 34.9% in the baseline method. Recall@10 was 85.1% compared to 60.7% in the baseline. When predicting links from articles to registrations, recall@1 was 83.4% for the classifier and 39.8% in the baseline. Recall@10 was 89.5% compared to 65.8% in the baseline. The proposed method improves on our baseline document similarity method to be feasible for identifying missing links in practice. Given a ClinicalTrials.gov registration, a user checking 10 ranked articles can expect to identify the matching article in at least 85% of cases, if the trial has been published. The proposed method can be used to improve the coupling of ClinicalTrials.gov and PubMed, with applications related to automating systematic review and evidence synthesis processes.

Clinical trial registration was associated with lower risk of bias compared with non-registered trials among trials included in systematic reviews

OBJECTIVE

To examine the association between clinical trial registration and risk of bias in clinical trials that have been included in systematic reviews. As a secondary objective, we evaluated the risk of bias among trials registered prospectively vs. retrospectively.

METHOD

Clinical trials published in 2005 or after included in a sample of 100 Cochrane systematic reviews published from 2014-2019.

RESULTS

Of 1,177 clinical trials identified, we verified 368 (31%) had been registered, of which 135 (36.7%) were registered prospectively (i.e., before or up to 1 month after enrollment of the first participant). Across the bias domains (one bias assessment for each domain per trial), the percentage of trials at low risk ranged from 29% to 58%; unclear risk ranged from to 26% to 61% and high risk ranged from 2% to 38%. Trials that had been registered had less high or unclear risk of bias in five domains: random sequence generation (univariate risk ratio [RR] 0.69, 95% confidence interval [95% CI] 0.58-0.81), allocation concealment (RR 0.64, 95% CI 0.57-0.72), performance bias (RR 0.65, 95% CI 0.58-0.72), detection bias (RR 0.70, 95% CI 0.62-0.78), and reporting bias (RR 0.62, 95% CI 0.53-0.73). An association between clinical trial registration and high or unclear risk of attrition bias could not be demonstrated nor refuted (RR 1.02, 95% CI 0.89-1.17). It also was observed in terms of overall risk of bias, that registered trials had less high or unclear overall risk of bias than trials that had not been registered (univariate RR 0.29, 95% CI 0.19-0.46). Prospective clinical trial registration was associated with low risks of selection bias due to inadequate allocation concealment, performance bias, and detection bias compared with retrospective clinical trial registration.

CONCLUSION

In a large sample of clinical trials included in recently published systematic reviews of interventions, clinical trial registration was associated with low risk of bias for five of the six domains examined.

A web-based tool for automatically linking clinical trials to their publications

OBJECTIVE

Evidence synthesis teams, physicians, policy makers, and patients and their families all have an interest in following the outcomes of clinical trials and would benefit from being able to evaluate both the results posted in trial registries and in the publications that arise from them. Manual searching for publications arising from a given trial is a laborious and uncertain process. We sought to create a statistical model to automatically identify PubMed articles likely to report clinical outcome results from each registered trial in ClinicalTrials.gov.

MATERIALS AND METHODS

A machine learning-based model was trained on pairs (publications known to be linked to specific registered trials). Multiple features were constructed based on the degree of matching between the PubMed article metadata and specific fields of the trial registry, as well as matching with the set of publications already known to be linked to that trial.

RESULTS

Evaluation of the model using known linked articles as gold standard showed that they tend to be top ranked (median best rank = 1.0), and 91% of them are ranked in the top 10.

DISCUSSION

Based on this model, we have created a free, public web-based tool that, given any registered trial in ClinicalTrials.gov, presents a ranked list of the PubMed articles in order of estimated probability that they report clinical outcome data from that trial. The tool should greatly facilitate studies of trial outcome results and their relation to the original trial designs.

The automation of relevant trial registration screening for systematic review updates: an evaluation study on a large dataset of ClinicalTrials.gov registrations

BACKGROUND

Clinical trial registries can be used as sources of clinical evidence for systematic review synthesis and updating. Our aim was to evaluate methods for identifying clinical trial registrations that should be screened for inclusion in updates of published systematic reviews.

METHODS

A set of 4644 clinical trial registrations (ClinicalTrials.gov) included in 1089 systematic reviews (PubMed) were used to evaluate two methods (document similarity and hierarchical clustering) and representations (L2-normalised TF-IDF, Latent Dirichlet Allocation, and Doc2Vec) for ranking 163,501 completed clinical trials by relevance. Clinical trial registrations were ranked for each systematic review using seeding clinical trials, simulating how new relevant clinical trials could be automatically identified for an update. Performance was measured by the number of clinical trials that need to be screened to identify all relevant clinical trials.

RESULTS

Using the document similarity method with TF-IDF feature representation and Euclidean distance metric, all relevant clinical trials for half of the systematic reviews were identified after screening 99 trials (IQR 19 to 491). The best-performing hierarchical clustering was using Ward agglomerative clustering (with TF-IDF representation and Euclidean distance) and needed to screen 501 clinical trials (IQR 43 to 4363) to achieve the same result.

CONCLUSION

An evaluation using a large set of mined links between published systematic reviews and clinical trial registrations showed that document similarity outperformed hierarchical clustering for identifying relevant clinical trials to include in systematic review updates.

Registration, publication, and outcome reporting among pivotal clinical trials that supported FDA approval of high-risk cardiovascular devices before and after FDAAA

BACKGROUND

Selective registration, publication, and outcome reporting of clinical trials distort the primary clinical evidence that is available to patients and clinicians regarding the safety and efficacy of US Food and Drug Administration (FDA)-approved medical devices. The purpose of this study is to compare registration, publication, and outcome reporting among pivotal clinical trials that supported FDA approval of high-risk (class III) cardiovascular devices before and after the FDA Amendment Act (FDAAA) was enacted in 2007.

METHODS

Using publicly available data from ClinicalTrials.gov , FDA summaries, and PubMed, we determined registration, publication, and reporting of findings for all pivotal clinical studies supporting FDA approval of new high-risk cardiovascular devices between 2005 and 2020, before and after FDAAA. For published studies, we compared both the primary efficacy outcome with the FDA's Premarket Approval (PMA) primary efficacy outcome and the published interpretation of findings with the FDA reviewer's interpretation (positive, equivocal, or negative).

RESULTS

Between 2005 and 2020, the FDA approved 156 high-risk cardiovascular devices on the basis of 165 pivotal trials, 48 (29%) of which were categorized as pre-FDAAA and 117 (71%) as post-FDAAA. Post-FDAAA, pivotal clinical trials were more likely to be registered (115 of 117 (98%) vs 24 of 48 (50%); p < 0.001), to report results (98 of 117 (87%) vs 7 of 48 (15%); p < 0.001) on ClinicalTrials.gov , and to be published (100 or 117 (85%) vs 28 of 48 (58%); p < 0.001) in peer-reviewed literature when compared to pre-FDAAA. Among published trials, rates of concordant primary efficacy outcome reporting were not significantly different between pre-FDAAA trials and post-FDAAA trials (24 of 28 (86%) vs 96 of 100 (96%); p = 0.07), nor were rates of concordant trial interpretation (27 of 28 (96%) vs 93 of 100 (93%); p = 0.44).

CONCLUSIONS

FDAAA was associated with increased registration, result reporting, and publication for trials supporting FDA approval of high-risk medical devices. Among published trials, rates of accurate primary efficacy outcome reporting and trial interpretation were high and no different post-FDAAA.

regCOVID: Tracking publications of registered COVID-19 studies

In response to the COVID-19 pandemic many clinical studies have been initiated leading to the need for efficient ways to track and analyze study results. We expanded our previous project that tracked registered COVID-19 clinical studies to also track result articles generated from these studies. We conducted searches of ClinicalTrials.gov and PubMed to identify articles linked to COVID-19 studies, and developed criteria based on the trial phase, intervention, location, and record recency to develop a prioritized list of result publications. We found 760 articles linked to 419 interventional trials (15.7% of all 2 669 COVID-19 interventional trials as of 15 August 2021), with 418 identified via abstract-link in PubMed and 342 via registry-link in ClinicalTrials.gov. Of the 419 trials publishing at least one article, 123 (29.4%) have multiple linked publications. We used an attention score to develop a prioritized list of all publications linked to COVID-19 trials and identified 58 publications that are result articles from late phase (Phase 3) trials with at least one US site and multiple study record updates. For COVID-19 vaccine trials, we found 69 linked result articles for 40 trials (13.9% of 290 total COVID-19 vaccine trials). Our method allows for the efficient identification of important COVID-19 articles that report results of registered clinical trials and are connected via a structured article-trial link.

New improved Aggregator: predicting which clinical trial articles derive from the same registered clinical trial

Objectives

To identify separate publications that report outcomes from the same underlying clinical trial, in order to avoid over-counting these as independent pieces of evidence.

Materials and Methods

We updated our previous model by creating larger, more recent, and more diverse positive and negative training sets consisting of article pairs that were (or not) linked to the same ClinicalTrials.gov trial registry number. Features were extracted from PubMed metadata; pairwise similarity scores were modeled using logistic regression and used to form clusters of articles that are likely to arise from the same registered clinical trial.

Results

Articles from the same trial were identified with high accuracy (F1 = 0.859), nominally better than the previous model (F1 = 0.843). Predicted clusters showed a low error rate of splitting of 8–11% (ie, when 2 articles belonged to the same trial but were assigned to different clusters). Performance was similar whether only randomized controlled trial articles or a more diverse set of clinical trial articles were processed.

Discussion

Metadata are surprisingly accurate in predicting when 2 articles derive from the same underlying clinical trial.

Conclusion

We have continued confidence in the Aggregator tool which can be accessed publicly at http://arrowsmith.psych.uic.edu/cgi-bin/arrowsmith_uic/RCT_Tagger.cgi.

Systematic reviews that include only published data may overestimate the effectiveness of analgesic medicines for low back pain: a systematic review and meta-analysis

OBJECTIVE

Systematic reviews of analgesics for low back pain generally include published data only. Obtaining data from unpublished trials is potentially important because they may impact effect sizes in meta-analyses. We determined whether including unpublished data from trial registries changes the effect sizes in meta-analyses of analgesics for low back pain.

STUDY DESIGN AND SETTING

Trial registries were searched for unpublished data that conformed to the inclusion criteria of n = 5 individual source systematic reviews. We reproduced the meta-analyses using data available from the original reviews and then reran the same analyses with the addition of new unpublished data.

RESULTS

Sixteen completed, unpublished, trials were eligible for inclusion in four of the source reviews. Data were available for five trials. We updated the analyses for two of the source reviews. The addition of data from two trials reduced the effect size of muscle relaxants, compared with sham, for recent-onset low back pain from -21.71 (95% CI: -28.23 to -15.19) to -2.34 (95% CI: -3.34 to -1.34) on a 0-100 scale for pain intensity. The addition of data from three trials (one enriched design) reduced the effect size of opioid analgesics, compared with sham, for chronic low back pain from -10.10 (95% CI: -12.81 to -7.39) to -9.31 (95% CI: -11.51 to -7.11). The effect reduced in the subgroup of enriched design studies, from -12.40 (95% CI: -16.90 to -7.91) to -11.34 (95% CI: -15.36 to -7.32), and in the subgroup of nonenriched design studies, from -7.27 (95% CI: -9.97 to -4.57) to -7.19 (95% CI: -9.24 to -5.14).

CONCLUSION

Systematic reviews should include reports of unpublished trials. The result for muscle relaxants conflicts with the conclusion of the published review and recent international guidelines. Adding unpublished data strengthens the evidence that opioid analgesics have small effects on persistent low back pain and more clearly suggests these effects may not be clinically meaningful.

Analysis of articles directly related to randomized trials finds poor protocol availability and inconsistent linking of articles

OBJECTIVES

Interpreting a randomized trial requires access to more than the main results paper. We aimed to determine the (1) proportion of trials referring to the protocol in the trial report and their accessibility, (2) proportion of protocols accessible from trial registry entry and by trial registration number search, and (3) types of additional publications associated with trial reports.

STUDY DESIGN AND SETTING

A previously gathered sample of randomized trials of nonpharmacological interventions published in 2009 was used. Trial reports and registry entries were searched for protocol mentions and obtained when possible. Related publications were identified using citation searching.

RESULTS

Of 133 trials, 96 (72%) mentioned the protocol within the report, 61 (64%) contained details about protocol acquisition, with 48 (36%) protocols obtainable. Of the 129 registered trials, 32 (25%) had protocols obtainable from registry entry. Citation tracking identified 1,030 related publications, most common were secondary analyses and qualitative studies.

CONCLUSION

Trial protocols facilitate good trial conduct and interpretation. However, they are often not linked to the main report or registry and can be difficult to obtain. Many trials have related publications that are inconsistently linked. Trial registries and registration numbers could facilitate the threading of articles related to a trial, but currently do not.

The clinical trials landscape for glioblastoma: is it adequate to develop new treatments?

Background

There have been few treatment advances for patients with glioblastoma (GBM) despite increasing scientific understanding of the disease. While factors such as intrinsic tumor biology and drug delivery are challenges to developing efficacious therapies, it is unclear whether the current clinical trial landscape is optimally evaluating new therapies and biomarkers.

Methods

We queried ClinicalTrials.gov for interventional clinical trials for patients with GBM initiated between January 2005 and December 2016 and abstracted data regarding phase, status, start and end dates, testing locations, endpoints, experimental interventions, sample size, clinical presentation/indication, and design to better understand the clinical trials landscape.

Results

Only approximately 8%-11% of patients with newly diagnosed GBM enroll on clinical trials with a similar estimate for all patients with GBM. Trial duration was similar across phases with median time to completion between 3 and 4 years. While 93% of clinical trials were in phases I-II, 26% of the overall clinical trial patient population was enrolled on phase III studies. Of the 8 completed phase III trials, only 1 reported positive results. Although 58% of the phase III trials were supported by phase II data with a similar endpoint, only 25% of these phase II trials were randomized.

Conclusions

The clinical trials landscape for GBM is characterized by long development times, inadequate dissemination of information, suboptimal go/no-go decision making, and low patient participation.

Non-invasive neuromodulation for migraine and cluster headache: a systematic review of clinical trials

Non-invasive neuromodulation therapies for migraine and cluster headache are a practical and safe alternative to pharmacologics. Comparisons of these therapies are difficult because of the heterogeneity in study designs. In this systematic review of clinical trials, the scientific rigour and clinical relevance of the available data were assessed to inform clinical decisions about non-invasive neuromodulation. PubMed, Cochrane Library and ClinicalTrials.gov databases and the WHO's International Clinical Trials Registry Platform were searched for relevant clinical studies of non-invasive neuromodulation devices for migraine and cluster headache (1 January 1990 to 31 January 2018), and 71 were identified. This analysis compared study designs using recommendations of the International Headache Society for pharmacological clinical trials, the only available guidelines for migraine and cluster headache. Non-invasive vagus nerve stimulation (nVNS), single-transcranial magnetic stimulation and external trigeminal nerve stimulation (all with regulatory clearance) were well studied compared with the other devices, for which studies frequently lacked proper blinding, sham controls and sufficient population sizes. nVNS studies demonstrated the most consistent adherence to available guidelines. Studies of all neuromodulation devices should strive to achieve the same high level of scientific rigour to allow for proper comparison across devices. Device-specific guidelines for migraine and cluster headache will be soon available, but adherence to current guidelines for pharmacological trials will remain a key consideration for investigators and clinicians.

Software engineering principles address current problems in the systematic review ecosystem

Systematic reviewers are simultaneously unable to produce systematic reviews fast enough to keep up with the availability of new trial evidence while overproducing systematic reviews that are unlikely to change practice because they are redundant or biased. Although the transparency and completeness of trial reporting has improved with changes in policy and new technologies, systematic reviews have not yet benefited from the same level of effort. We found that new methods and tools used to automate aspects of systematic review processes have focused on improving the efficiency of individual systematic reviews rather than the efficiency of the entire ecosystem of systematic review production. We use software engineering principles to review challenges and opportunities for improving the interoperability, integrity, efficiency, and maintainability. We conclude by recommending ways to improve access to structured systematic review results. Major opportunities for improving systematic reviews will come from new tools and changes in policy focused on doing the right systematic reviews rather than just doing more of them faster.

Therapy for glioblastoma: is it working?

Glioblastoma (GBM) remains one of the most intransigent of cancers, with a median overall survival of only 15 months after diagnosis. Drug treatments have largely proven ineffective; it is thought that this is related to the heterogeneous nature and plasticity of GBM-initiating stem cell lineages. Although many combination drug therapies are being positioned to address tumour heterogeneity, the most promising therapeutic approaches for GBM to date appear to be those targeting GBM by vaccination or antibody- and cell-based immunotherapy. We review the most recent clinical trials for GBM and discuss the role of adaptive clinical trials in developing personalised treatment strategies to address intra- and inter-tumoral heterogeneity.

Tracking the timely dissemination of clinical studies. Characteristics and impact of 10 tracking variables

Background: Several meta-research studies and benchmarking activities have assessed how comprehensively and timely, academic institutions and private companies publish their clinical studies. These current "clinical trial tracking" activities differ substantially in how they sample relevant studies, and how they follow up on their publication. Methods: To allow informed policy and decision making on future publication assessment and benchmarking of institutions and companies, this paper outlines and discusses 10 variables that influence the tracking of timely publications. Tracking variables were initially selected by experts and by the authors through discussion. To validate the completeness of our set of variables, we conducted i) an explorative review of tracking studies and ii) an explorative tracking of registered clinical trials of three leading German university medical centres. Results: We identified the following 10 relevant variables impacting the tracking of clinical studies: 1) responsibility for clinical studies, 2) type and characteristics of clinical studies, 3) status of clinical studies, 4) source for sampling, 5) timing of registration, 6) determination of completion date, 7) timeliness of dissemination, 8) format of dissemination, 9) source for tracking, and 10) inter-rater reliability. Based on the description of these tracking variables and their influence, we discuss which variables could serve in what ways as a standard assessment of "timely publication". Conclusions: To facilitate the tracking and consequent benchmarking of how often and how timely academic institutions and private companies publish clinical study results, we have two core recommendations. First, the improvement in the link between registration and publication, for example via institutional policies for academic institutions and private companies. Second, the comprehensive and transparent reporting of tracking studies according to the 10 variables presented in this paper.

Registration of published randomized trials: a systematic review and meta-analysis

BACKGROUND

Prospective trial registration is a powerful tool to prevent reporting bias. We aimed to determine the extent to which published randomized controlled trials (RCTs) were registered and registered prospectively.

METHODS

We searched MEDLINE and EMBASE from January 2005 to October 2017; we also screened all articles cited by or citing included and excluded studies, and the reference lists of related reviews. We included studies that examined published RCTs and evaluated their registration status, regardless of medical specialty or language. We excluded studies that assessed RCT registration status only through mention of registration in the published RCT, without searching registries or contacting the trial investigators. Two independent reviewers blinded to the other's work performed the selection. Following PRISMA guidelines, two investigators independently extracted data, with discrepancies resolved by consensus. We calculated pooled proportions and 95% confidence intervals using random-effects models.

RESULTS

We analyzed 40 studies examining 8773 RCTs across a wide range of clinical specialties. The pooled proportion of registered RCTs was 53% (95% confidence interval 44% to 58%), with considerable between-study heterogeneity. A subset of 24 studies reported data on prospective registration across 5529 RCTs. The pooled proportion of prospectively registered RCTs was 20% (95% confidence interval 15% to 25%). Subgroup analyses showed that registration was higher for industry-supported and larger RCTs. A meta-regression analysis across 19 studies (5144 RCTs) showed that the proportion of registered trials significantly increased over time, with a mean proportion increase of 27%, from 25 to 52%, between 2005 and 2015.

CONCLUSIONS

The prevalence of trial registration has increased over time, but only one in five published RCTs is prospectively registered, undermining the validity and integrity of biomedical research.

Trial registration in pediatric surgery trials

BACKGROUND

Prospective clinical trial registration serves to increase transparency and to mitigate selective reporting bias. An assessment of adult surgical trials revealed poor trial registration practice with incomplete provision of information in registries and inconsistent information in the corresponding publication. The extent and completeness of pediatric surgical trial registration are unknown. We aimed to determine the proportion and adequacy of clinical trial registration in pediatric surgery trials published in 2014.

METHODS

Using sensitive search strategies in MEDLINE, abstracts and full-texts of prospective pediatric intervention studies published in 2014 were screened in duplicate. Pediatric surgical trials were included. Clinical trial registration numbers obtained from publications were searched in trial registries. Data were extracted based on WHO 20-item minimum data set to determine the completeness of registration data. The proportion of registered trials was recorded and registration data were compared to reported data in the corresponding publication.

RESULTS

Our search and abstract screening identified 3375 articles for full text review. Following coding, a total of 54 pediatric surgical trials were included and analyzed; 28% (15/54) of which published a registration number. In trials which reported a registration number, 40% (6/15) were retrospectively registered and 40% (6/15) had made changes to their registered primary and/or secondary outcome measures. One included published trial reported an incorrect registration number.

CONCLUSIONS

Analysis of pediatric surgery trials published in 2014 revealed a poor prospective trial registration rate and incomplete registration data. Our study supports future initiatives for improved registration behaviors in pediatric surgery trials to ensure high-quality, transparent, reproducible evidence is generated.

STUDY TYPE

Therapeutic (clinical trials), level II.

Clinical trial transparency update: an assessment of the disclosure of results of company-sponsored trials associated with new medicines approved in Europe in 2014

BACKGROUND

The objective of this study was to assess the timely disclosure of results of company-sponsored clinical trials related to all new medicines approved by the European Medicines Agency (EMA) during 2014. This is the final extension of three previously reported studies of trials related to all new medicines approved in Europe in 2009, 2010 and 2011, and in 2012 and 2013. The original study found that over a three-year period over three-quarters of all trials were disclosed within 12 months and almost 90% were disclosed by the end of the study (31 January 2013). The extension studies (2012 and 2013 approvals) both showed an improvement in results disclosure within 12 months to 90%, and an overall disclosure rate of 92% and 93% respectively by the end of the studies.

METHODS

The methodology used was exactly as previously reported. Various publicly available information sources were searched for both clinical trial registration and disclosure of results. All completed company-sponsored trials related to each new medicine approved for marketing by the EMA in 2014, carried out in patients and recorded on a clinical trials registry and/or included in an EMA European Public Assessment Report (EPAR), were included. Information sources were searched between 1 May and 31 July 2016.

OUTCOME MEASURES AND RESULTS

The main outcome measure was the proportion of trials for which results had been disclosed on a registry or in the scientific literature either within 12 months of the later of either first regulatory approval or trial completion, or by 31 July 2016 (end of survey). Of the completed trials associated with 32 new medicines licensed to 22 different companies in 2014, results of 93% (505/542) had been disclosed within 12 months, and results of 96% (518/542) had been disclosed by 31 July 2016.

CONCLUSIONS

The disclosure rate within 12 months of 93% suggests that industry is continuing to achieve disclosure in a timely manner. The overall disclosure rate at study end of 96% indicates that the improvement in transparency amongst company-sponsored trials has been maintained in the trials associated with new medicines approved in 2014.

Automatically Linking Registered Clinical Trials to their Published Results with Deep Highway Networks

As medical science continues to advance, health care professionals and researchers are increasingly turning to clinical trials to obtain evidence supporting best-practice treatment options. While clinical trial registries such as Clinical-Trials.gov aim to facilitate these needs, it has been shown that many trials in the registry do not contain links to their published results. To address this problem, we present NCT Link, a system for automatically linking registered clinical trials to published MEDLINE articles reporting their results. NCT Link incorporates state-of-the-art deep learning and information retrieval techniques by automatically learning a Deep Highway Network (DHN) that estimates the likelihood that a MEDLINE article reports the results of a clinical trial. Our experimental results indicate that NCT Link obtains 30%-58% improved performance over previously reported automatic systems, suggesting that NCT Link could become a valuable tool for health care providers seeking to deliver best-practice medical care informed by evidence of clinical trials as well as (a) researchers investigating selective publication and reporting of clinical trial outcomes, and (b) study designers seeking to avoid unnecessary duplication of research efforts.

A shared latent space matrix factorisation method for recommending new trial evidence for systematic review updates

BACKGROUND

Clinical trial registries can be used to monitor the production of trial evidence and signal when systematic reviews become out of date. However, this use has been limited to date due to the extensive manual review required to search for and screen relevant trial registrations. Our aim was to evaluate a new method that could partially automate the identification of trial registrations that may be relevant for systematic review updates.

MATERIALS AND METHODS

We identified 179 systematic reviews of drug interventions for type 2 diabetes, which included 537 clinical trials that had registrations in ClinicalTrials.gov. Text from the trial registrations were used as features directly, or transformed using Latent Dirichlet Allocation (LDA) or Principal Component Analysis (PCA). We tested a novel matrix factorisation approach that uses a shared latent space to learn how to rank relevant trial registrations for each systematic review, comparing the performance to document similarity to rank relevant trial registrations. The two approaches were tested on a holdout set of the newest trials from the set of type 2 diabetes systematic reviews and an unseen set of 141 clinical trial registrations from 17 updated systematic reviews published in the Cochrane Database of Systematic Reviews. The performance was measured by the number of relevant registrations found after examining 100 candidates (recall@100) and the median rank of relevant registrations in the ranked candidate lists.

RESULTS

The matrix factorisation approach outperformed the document similarity approach with a median rank of 59 (of 128,392 candidate registrations in ClinicalTrials.gov) and recall@100 of 60.9% using LDA feature representation, compared to a median rank of 138 and recall@100 of 42.8% in the document similarity baseline. In the second set of systematic reviews and their updates, the highest performing approach used document similarity and gave a median rank of 67 (recall@100 of 62.9%).

CONCLUSIONS

A shared latent space matrix factorisation method was useful for ranking trial registrations to reduce the manual workload associated with finding relevant trials for systematic review updates. The results suggest that the approach could be used as part of a semi-automated pipeline for monitoring potentially new evidence for inclusion in a review update.

Comparison of reporting phase III randomized controlled trials of antibiotic treatment for common bacterial infections in ClinicalTrials.gov and matched publications

OBJECTIVES

Discrepancies between ClinicalTrials.gov entries and matching publications were previously described in general medicine. We aimed to evaluate the consistency of reporting in trials addressing systemic antibiotic therapy.

METHODS

We searched ClinicalTrials.gov for completed phase III trials comparing antibiotic regimens until May 2017. Matched publications were identified in PubMed. Two independent reviewers extracted data and identified inconsistencies. Reporting was assessed among studies started before and after 1 July 2005, when the International Committee of Medical Journal Editors (ICMJE) required mandatory registration as a prerequisite for considering a trial for publication.

RESULTS

Matching publications were identified for 75 (70%) of 107 ClinicalTrials.gov entries. Median time from study completion to publication was 26 months (interquartile range 19-42). Primary outcome definition was inconsistent between ClinicalTrials.gov and publications in seven trials (7/72, 10%) and reporting of the primary outcome timeframe was inconsistent in 14 (14/71, 20%). Secondary outcomes definitions were inconsistent in 36 trials (36/66, 55%). Reporting of inclusion criteria and study timeline were inconsistent in 17% (13/65) and 3% (2/65), respectively. Trials started after July 2005 were significantly less likely to have reporting inconsistencies and were published in higher impact factor journals.

CONCLUSIONS

We found a lower inconsistency rate of outcome reporting compared with other medical disciplines. Reporting completeness and consistency were significantly better after July 2005. The ICMJE requirement for mandatory registration was associated with significant improvement in reporting quality in infectious diseases trials. Prolonged time lag to publication and missing data from unpublished trials should raise a discussion on current reporting and publishing procedures.

Unreported links between trial registrations and published articles were identified using document similarity measures in a cross-sectional analysis of ClinicalTrials.gov

OBJECTIVES

Trial registries can be used to measure reporting biases and support systematic reviews, but 45% of registrations do not provide a link to the article reporting on the trial. We evaluated the use of document similarity methods to identify unreported links between ClinicalTrials.gov and PubMed.

STUDY DESIGN AND SETTING

We extracted terms and concepts from a data set of 72,469 ClinicalTrials.gov registrations and 276,307 PubMed articles and tested methods for ranking articles across 16,005 reported links and 90 manually identified unreported links. Performance was measured by the median rank of matching articles and the proportion of unreported links that could be found by screening ranked candidate articles in order.

RESULTS

The best-performing concept-based representation produced a median rank of 3 (interquartile range [IQR] 1-21) for reported links and 3 (IQR 1-19) for the manually identified unreported links, and term-based representations produced a median rank of 2 (1-20) for reported links and 2 (IQR 1-12) in unreported links. The matching article was ranked first for 40% of registrations, and screening 50 candidate articles per registration identified 86% of the unreported links.

CONCLUSION

Leveraging the growth in the corpus of reported links between ClinicalTrials.gov and PubMed, we found that document similarity methods can assist in the identification of unreported links between trial registrations and corresponding articles.

Practical guidance for using multiple data sources in systematic reviews and meta-analyses (with examples from the MUDS study)

Data for individual trials included in systematic reviews may be available in multiple sources. For example, a single trial might be reported in 2 journal articles and 3 conference abstracts. Because of differences across sources, source selection can influence the results of systematic reviews. We used our experience in the Multiple Data Sources in Systematic Reviews (MUDS) study, and evidence from previous studies, to develop practical guidance for using multiple data sources in systematic reviews. We recommend the following: (1) Specify which sources you will use. Before beginning a systematic review, consider which sources are likely to contain the most useful data. Try to identify all relevant reports and to extract information from the most reliable sources. (2) Link individual trials with multiple sources. Write to authors to determine which sources are likely related to the same trials. Use a modified Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) flowchart to document both the selection of trials and the selection of sources. (3) Follow a prespecified protocol for extracting trial characteristics from multiple sources. Identify differences among sources, and contact study authors to resolve differences if possible. (4) Prespecify outcomes and results to examine in the review and meta-analysis. In your protocol, describe how you will handle multiple outcomes within each domain of interest. Look for outcomes using all eligible sources. (5) Identify which data sources were included in the review. Consider whether the results might have been influenced by data sources used. (6) To reduce bias, and to reduce research waste, share the data used in your review.

Study-based registers of randomized controlled trials: Starting a systematic review with data extraction or meta-analysis

Introduction: Despite years of use of study-based registers for storing reports of randomized controlled trials (RCTs), the methodology used in developing such registers/databases has not been documented. Such registers are integral to the process of scientific reviewing. We document and discuss methodological aspects of the development and use of study-based registers. Although the content is focused on the study-based register of randomized/controlled clinical trials, this work applies to developers of databases of all sorts of studies related to the human, animals, cells, genes, and molecules. Methods: We describe necessity, rationale, and steps for the development, utilization and maintenance of study-based registers as well as the challenges and gains for the organizations supporting systematic reviews of the published and unpublished literature. Conclusion: The ultimate goal of having a study-based register is to facilitate efficient production of systematic reviews providing rapid, yet accurate, evidence for the decision-makers. We argue that moving towards study-based registers is an inevitable welcome direction and that infrastructures are ready for such movement.