Reanalyses of randomized clinical trial data

Estimating the replicability of highly cited clinical research (2004-2018)

INTRODUCTION

Previous studies about the replicability of clinical research based on the published literature have suggested that highly cited articles are often contradicted or found to have inflated effects. Nevertheless, there are no recent updates of such efforts, and this situation may have changed over time.

METHODS

We searched the Web of Science database for articles studying medical interventions with more than 2000 citations, published between 2004 and 2018 in high-impact medical journals. We then searched for replications of these studies in PubMed using the PICO (Population, Intervention, Comparator and Outcome) framework. Replication success was evaluated by the presence of a statistically significant effect in the same direction and by overlap of the replication's effect size confidence interval (CIs) with that of the original study. Evidence of effect size inflation and potential predictors of replicability were also analyzed.

RESULTS

A total of 89 eligible studies, of which 24 had valid replications (17 meta-analyses and 7 primary studies) were found. Of these, 21 (88%) had effect sizes with overlapping CIs. Of 15 highly cited studies with a statistically significant difference in the primary outcome, 13 (87%) had a significant effect in the replication as well. When both criteria were considered together, the replicability rate in our sample was of 20 out of 24 (83%). There was no evidence of systematic inflation in these highly cited studies, with a mean effect size ratio of 1.03 [95% CI (0.88, 1.21)] between initial and subsequent effects. Due to the small number of contradicted results, our analysis had low statistical power to detect predictors of replicability.

CONCLUSION

Although most studies did not have eligible replications, the replicability rate of highly cited clinical studies in our sample was higher than in previous estimates, with little evidence of systematic effect size inflation. This estimate is based on a very select sample of studies and may not be generalizable to clinical research in general.

Riluzole for Degenerative Cervical Myelopathy: A Secondary Analysis of the CSM-PROTECT Trial

Importance

The modified Japanese Orthopaedic Association (mJOA) scale is the most common scale used to represent outcomes of degenerative cervical myelopathy (DCM); however, it lacks consideration for neck pain scores and neglects the multidimensional aspect of recovery after surgery.

Objective

To use a global statistical approach that incorporates assessments of multiple outcomes to reassess the efficacy of riluzole in patients undergoing spinal surgery for DCM.

Design, Setting, and Participants

This was a secondary analysis of prespecified secondary end points within the Efficacy of Riluzole in Surgical Treatment for Cervical Spondylotic Myelopathy (CSM-PROTECT) trial, a multicenter, double-blind, phase 3 randomized clinical trial conducted from January 2012 to May 2017. Adult surgical patients with DCM with moderate to severe myelopathy (mJOA scale score of 8-14) were randomized to receive either riluzole or placebo. The present study was conducted from July to December 2023.

Intervention

Riluzole (50 mg twice daily) or placebo for a total of 6 weeks, including 2 weeks prior to surgery and 4 weeks following surgery.

Main Outcomes and Measures

The primary outcome measure was a difference in clinical improvement from baseline to 1-year follow-up, assessed using a global statistical test (GST). The 36-Item Short Form Health Survey Physical Component Score (SF-36 PCS), arm and neck pain numeric rating scale (NRS) scores, American Spinal Injury Association (ASIA) motor score, and Nurick grade were combined into a single summary statistic known as the global treatment effect (GTE).

Results

Overall, 290 patients (riluzole group, 141; placebo group, 149; mean [SD] age, 59 [10.1] years; 161 [56%] male) were included. Riluzole showed a significantly higher probability of global improvement compared with placebo at 1-year follow-up (GTE, 0.08; 95% CI, 0.00-0.16; P = .02). A similar favorable global response was seen at 35 days and 6 months (GTE for both, 0.07; 95% CI, -0.01 to 0.15; P = .04), although the results were not statistically significant. Riluzole-treated patients had at least a 54% likelihood of achieving better outcomes at 1 year compared with the placebo group. The ASIA motor score and neck and arm pain NRS combination at 1 year provided the best-fit parsimonious model for detecting a benefit of riluzole (GTE, 0.11; 95% CI, 0.02-0.16; P = .007).

Conclusions and Relevance

In this secondary analysis of the CSM-PROTECT trial using a global outcome technique, riluzole was associated with improved clinical outcomes in patients with DCM. The GST offered probability-based results capable of representing diverse outcome scales and should be considered in future studies assessing spine surgery outcomes.

Initial data analysis for longitudinal studies to build a solid foundation for reproducible analysis

Initial data analysis (IDA) is the part of the data pipeline that takes place between the end of data retrieval and the beginning of data analysis that addresses the research question. Systematic IDA and clear reporting of the IDA findings is an important step towards reproducible research. A general framework of IDA for observational studies includes data cleaning, data screening, and possible updates of pre-planned statistical analyses. Longitudinal studies, where participants are observed repeatedly over time, pose additional challenges, as they have special features that should be taken into account in the IDA steps before addressing the research question. We propose a systematic approach in longitudinal studies to examine data properties prior to conducting planned statistical analyses. In this paper we focus on the data screening element of IDA, assuming that the research aims are accompanied by an analysis plan, meta-data are well documented, and data cleaning has already been performed. IDA data screening comprises five types of explorations, covering the analysis of participation profiles over time, evaluation of missing data, presentation of univariate and multivariate descriptions, and the depiction of longitudinal aspects. Executing the IDA plan will result in an IDA report to inform data analysts about data properties and possible implications for the analysis plan-another element of the IDA framework. Our framework is illustrated focusing on hand grip strength outcome data from a data collection across several waves in a complex survey. We provide reproducible R code on a public repository, presenting a detailed data screening plan for the investigation of the average rate of age-associated decline of grip strength. With our checklist and reproducible R code we provide data analysts a framework to work with longitudinal data in an informed way, enhancing the reproducibility and validity of their work.

Machine learning-based prediction models for pressure injury: A systematic review and meta-analysis

Despite the fact that machine learning (ML) algorithms to construct predictive models for pressure injury development are widely reported, the performance of the model remains unknown. The goal of the review was to systematically appraise the performance of ML models in predicting pressure injury. PubMed, Embase, Cochrane Library, Web of Science, CINAHL, Grey literature and other databases were systematically searched. Original journal papers were included which met the inclusion criteria. The methodological quality was assessed independently by two reviewers using the Prediction Model Risk of Bias Assessment Tool (PROBAST). Meta-analysis was performed with Metadisc software, with the area under the receiver operating characteristic curve, sensitivity and specificity as effect measures. Chi-squared and I2 tests were used to assess the heterogeneity. A total of 18 studies were included for the narrative review, and 14 of them were eligible for meta-analysis. The models achieved excellent pooled AUC of 0.94, sensitivity of 0.79 (95% CI [0.78-0.80]) and specificity of 0.87 (95% CI [0.88-0.87]). Meta-regressions did not provide evidence that model performance varied by data or model types. The present findings indicate that ML models show an outstanding performance in predicting pressure injury. However, good-quality studies should be conducted to verify our results and confirm the clinical value of ML in pressure injury development.

Evaluating the Utility and Privacy of Synthetic Breast Cancer Clinical Trial Data Sets

PURPOSE

There is strong interest from patients, researchers, the pharmaceutical industry, medical journal editors, funders of research, and regulators in sharing clinical trial data for secondary analysis. However, data access remains a challenge because of concerns about patient privacy. It has been argued that synthetic data generation (SDG) is an effective way to address these privacy concerns. There is a dearth of evidence supporting this on oncology clinical trial data sets, and on the utility of privacy-preserving synthetic data. The objective of the proposed study is to validate the utility and privacy risks of synthetic clinical trial data sets across multiple SDG techniques.

METHODS

We synthesized data sets from eight breast cancer clinical trial data sets using three types of generative models: sequential synthesis, conditional generative adversarial network, and variational autoencoder. Synthetic data utility was evaluated by replicating the published analyses on the synthetic data and assessing concordance of effect estimates and CIs between real and synthetic data. Privacy was evaluated by measuring attribution disclosure risk and membership disclosure risk.

RESULTS

Utility was highest using the sequential synthesis method where all results were replicable and the CI overlap most similar or higher for seven of eight data sets. Both types of privacy risks were low across all three types of generative models.

DISCUSSION

Synthetic data using sequential synthesis methods can act as a proxy for real clinical trial data sets, and simultaneously have low privacy risks. This type of generative model can be one way to enable broader sharing of clinical trial data.

Quality of life, patient satisfaction, and cardiovascular outcomes of the randomised 2 x 3 factorial Copenhagen insulin and Metformin therapy (CIMT) trial - A detailed statistical analysis plan

Background

The evidence on the effects of metformin and insulin in type 2 diabetes patients on quality of life, patient satisfaction, and cardiovascular outcomes is unclear.

Methods

The Copenhagen Insulin and Metformin Therapy (CIMT) trial is an investigator-initiated multicentre, randomised, placebo-controlled trial with a 2 × 3 factorial design conducted at eight hospitals in Denmark. Participants with type 2 diabetes were randomised to metformin (n = 206) versus placebo (n = 206); in combination with open-label biphasic insulin aspart one to three times daily (n = 137) versus insulin aspart three times daily in combination with insulin detemir once daily (n = 138) versus insulin detemir once daily (n = 137).We present a detailed description of the methodology and statistical analysis of the clinical CIMT outcomes including a detailed description of tests of the assumptions behind the statistical analyses. The outcomes are quality of life (Short Form Health Survey (SF-36)), Diabetes Medication Satisfaction Questionnaire, and Insulin Treatment Satisfaction Questionnaire (assessed at entry and 18 months after randomisation) and cardiovascular outcomes including time to a composite of either myocardial infarction, stroke, peripheral amputation, coronary revascularisation, peripheral revascularisation, or death.

Discussions

This statistical analysis plan ensure the highest possible quality of the subsequent post-hoc analyses.

Trial registration

The protocol was approved by the Regional Committee on Biomedical Research Ethics (H-D-2007-112), the Danish Medicines Agency (EudraCT: 2007-006665-33 CIMT), and registered within ClinicalTrials.gov (NCT00657943, 8^th of April 2008).

Emulation of Randomized Clinical Trials With Nonrandomized Database Analyses: Results of 32 Clinical Trials

Importance

Nonrandomized studies using insurance claims databases can be analyzed to produce real-world evidence on the effectiveness of medical products. Given the lack of baseline randomization and measurement issues, concerns exist about whether such studies produce unbiased treatment effect estimates.

Objective

To emulate the design of 30 completed and 2 ongoing randomized clinical trials (RCTs) of medications with database studies using observational analogues of the RCT design parameters (population, intervention, comparator, outcome, time [PICOT]) and to quantify agreement in RCT-database study pairs.

Design, Setting, and Participants

New-user cohort studies with propensity score matching using 3 US claims databases (Optum Clinformatics, MarketScan, and Medicare). Inclusion-exclusion criteria for each database study were prespecified to emulate the corresponding RCT. RCTs were explicitly selected based on feasibility, including power, key confounders, and end points more likely to be emulated with real-world data. All 32 protocols were registered on ClinicalTrials.gov before conducting analyses. Emulations were conducted from 2017 through 2022.

Exposures

Therapies for multiple clinical conditions were included.

Main Outcomes and Measures

Database study emulations focused on the primary outcome of the corresponding RCT. Findings of database studies were compared with RCTs using predefined metrics, including Pearson correlation coefficients and binary metrics based on statistical significance agreement, estimate agreement, and standardized difference.

Results

In these highly selected RCTs, the overall observed agreement between the RCT and the database emulation results was a Pearson correlation of 0.82 (95% CI, 0.64-0.91), with 75% meeting statistical significance, 66% estimate agreement, and 75% standardized difference agreement. In a post hoc analysis limited to 16 RCTs with closer emulation of trial design and measurements, concordance was higher (Pearson r, 0.93; 95% CI, 0.79-0.97; 94% meeting statistical significance, 88% estimate agreement, 88% standardized difference agreement). Weaker concordance occurred among 16 RCTs for which close emulation of certain design elements that define the research question (PICOT) with data from insurance claims was not possible (Pearson r, 0.53; 95% CI, 0.00-0.83; 56% meeting statistical significance, 50% estimate agreement, 69% standardized difference agreement).

Conclusions and Relevance

Real-world evidence studies can reach similar conclusions as RCTs when design and measurements can be closely emulated, but this may be difficult to achieve. Concordance in results varied depending on the agreement metric. Emulation differences, chance, and residual confounding can contribute to divergence in results and are difficult to disentangle.

Evidence for the carbohydrate-insulin model in a reanalysis of the Diet Intervention Examining The Factors Interacting with Treatment Success (DIETFITS) trial

BACKGROUND

The Diet Intervention Examining The Factors Interacting with Treatment Success (DIETFITS) trial demonstrated that meaningful weight loss can be achieved with either a "healthy low-carbohydrate diet" (LCD) or "healthy low-fat diet" (LFD). However, because both diets substantially decreased glycemic load (GL), the dietary factors mediating weight loss remain unclear.

OBJECTIVES

We aimed to explore the contribution of macronutrients and GL to weight loss in DIETFITS and examine a hypothesized relationship between GL and insulin secretion.

DESIGN

This study is a secondary data analysis of the DIETFITS trial, in which participants with overweight or obesity (aged 18-50 y) were randomized to a 12-mo LCD (N = 304) or LFD (N = 305).

RESULTS

Measures related to carbohydrate intake (total amount, glycemic index, added sugar, and fiber) showed strong associations with weight loss at 3-, 6-, and 12-mo time points in the full cohort, whereas those related to total fat intake showed weak to no associations. A biomarker of carbohydrate (triglyceride/HDL cholesterol ratio) predicted weight loss at all time points (3-mo: β [kg/biomarker z-score change] = 1.1, P = 3.5 × 10^-9; 6-mo: β = 1.7, P = 1.1 × 10^-9; and 12-mo: β = 2.6, P = 1.5 × 10^-15), whereas that of fat (low-density lipoprotein cholesterol + HDL cholesterol) did not (all time points: P = NS). In a mediation model, GL explained most of the observed effect of total calorie intake on weight change. Dividing the cohort into quintiles of baseline insulin secretion and GL reduction revealed evidence of effect modification for weight loss, with P = 0.0009 at 3 mo, P = 0.01 at 6 mo, and P = 0.07 at 12 mo.

CONCLUSIONS

As predicted by the carbohydrate-insulin model of obesity, weight loss in both diet groups of DIETFITS seems to have been driven by the reduction of GL more so than dietary fat or calories, an effect that may be most pronounced among those with high insulin secretion. These findings should be interpreted cautiously in view of the exploratory nature of this study.

TRIAL REGISTRATION

ClinicalTrials.gov (NCT01826591).

Restoring Study PRGF: a randomized clinical trial on plasma rich in growth factors for knee osteoarthritis

BACKGROUND

A randomized clinical trial assessing plasma rich in growth factors (PRGF) versus hyaluronic acid for knee osteoarthritis was published in 2012 (sponsor trial ID BTI-01-EC/07/ART). Evidence of misreporting was discovered following access to unpublished materials. In accordance with the principles of the Restoring Invisible and Abandoned Trials (RIAT) initiative, we sought to re-analyse Study PRGF based on the unpublished trial materials.

METHODS

Reanalysis was made possible primarily based on two unpublished study documents (original trial protocol and final report) obtained from the authors of the original publication. A call to action, calling on the authors to correct the original publication, was publicly issued. The involved ethics committee was repeatedly approached and extensive discussion with the authors ensued. After no agreement to correct the paper was reached, we embarked on this restoration. Reanalysis was focused on providing updated analyses for efficacy and safety.

RESULTS

The efficacy of PRGF was not statistically different from hyaluronic acid for any prespecified primary or secondary efficacy outcomes. For the primary endpoint, the percent of patients on PRGF compared to hyaluronic acid with a decrease >40% in WOMAC pain subscale score was 5.4% higher; 95% confidence interval (CI) -10.4% to 21.3%; p = 0.505. This differs from the original publication that reported a non-prespecified primary endpoint (decrease >50% in WOMAC pain subscale score) which was 14.1% higher; 95% CI 0.5 to 27.6%; p=0.044. Furthermore, in contrast to the article statement that all the adverse events disappeared in 48 h, at least two patients in the hyaluronic arm and five patients in the PRGF arm reported persistent adverse events. Inadequate disclosure of conflicts of interest in the original publication was also noted.

CONCLUSIONS

This reanalysis of Study PRGF found no clinically or statistically significant benefit from PRGF compared to hyaluronic acid. The restoration of Study PRGF shows the urgency of important changes to trial reporting and oversight practices. In the future, timely access to all clinical trial documents is needed to minimize the risk of reporting bias. Similarly, ethics committees should be ready to intervene whenever a case of potential misconduct arises.

TRIAL REGISTRATION

This is a RIAT project, whose original trial was approved and registered on 19 December 2007 by the Ethics Committee of the Basque Country, Spain, as BTI-01-EC/07/ART.

Machine learning models for predicting acute kidney injury: a systematic review and critical appraisal

BACKGROUND

The number of studies applying machine learning (ML) to predict acute kidney injury (AKI) has grown steadily over the past decade. We assess and critically appraise the state of the art in ML models for AKI prediction, considering performance, methodological soundness, and applicability.

METHODS

We searched PubMed and ArXiv, extracted data, and critically appraised studies based on the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD), Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS), and Prediction Model Risk of Bias Assessment Tool (PROBAST) guidelines.

RESULTS

Forty-six studies from 3166 titles were included. Thirty-eight studies developed a model, five developed and externally validated one, and three studies externally validated one. Flexible ML methods were used more often than deep learning, although the latter was common with temporal variables and text as predictors. Predictive performance showed an area under receiver operating curves ranging from 0.49 to 0.99. Our critical appraisal identified a high risk of bias in 39 studies. Some studies lacked internal validation, whereas external validation and interpretability of results were rarely considered. Fifteen studies focused on AKI prediction in the intensive care setting, and the US-derived Medical Information Mart for Intensive Care (MIMIC) data set was commonly used. Reproducibility was limited as data and code were usually unavailable.

CONCLUSIONS

Flexible ML methods are popular for the prediction of AKI, although more complex models based on deep learning are emerging. Our critical appraisal identified a high risk of bias in most models: Studies should use calibration measures and external validation more often, improve model interpretability, and share data and code to improve reproducibility.

Testosterone Serum Levels Are Related to Sperm DNA Fragmentation Index Reduction after FSH Administration in Males with Idiopathic Infertility

Purpose: Although a robust physiological rationale supports follicle stimulating hormone (FSH) use in male idiopathic infertility, useful biomarkers to evaluate its efficacy are not available. Thus, the primary aim of the study was to evaluate if testosterone serum levels are related to sperm DNA fragmentation (sDF) index change after FSH administration. The secondary aim was to confirm sDF index validity as a biomarker of FSH administration effectiveness in male idiopathic infertility. Methods: A retrospective, post-hoc re-analysis was performed on prospectively collected raw data of clinical trials in which idiopathic infertile men were treated with FSH and both testosterone serum levels and sDF were reported. Results: Three trials were included, accounting for 251 patients. The comprehensive analysis confirmed FSH’s beneficial effect on spermatogenesis detected in each trial. Indeed, an overall significant sDF decrease (p < 0.001) of 20.2% of baseline value was detected. Although sDF resulted to be unrelated to testosterone serum levels at baseline, a significant correlation was highlighted after three months of FSH treatment (p = 0.002). Moreover, testosterone serum levels and patients’ age significantly correlated with sDF (p = 0.006). Dividing the cohort into responders/not responders to FSH treatment according to sDF change, the FSH effectiveness in terms of sDF improvement was related to testosterone and male age (p = 0.003). Conclusion: Exogenous FSH administration in male idiopathic infertility is efficient in reducing sDF basal levels by about 20%. In terms of sDF reduction, 59.2% of the patients treated were FSH-responders. After three months of FSH administration, a significant inverse correlation between sDF and testosterone was detected, suggesting an association between the FSH-administration-related sDF improvement and testosterone serum levels increase. These observations lead to the hypothesis that FSH may promote communications or interactions between Sertoli cells and Leydig cells.

Many researchers were not compliant with their published data sharing statement: a mixed-methods study

OBJECTIVES

The objective of the study was to analyze researchers' compliance with their data availability statement (DAS) from manuscripts published in open-access journals with the mandatory DAS.

STUDY DESIGN AND SETTING

We analyzed all articles from 333 open-access journals published during January 2019 by BioMed Central. We categorized types of the DAS. We surveyed corresponding authors who wrote in the DAS that they would share the data. Consent to participate in the study was sought for all included manuscripts. After accessing raw data sets, we checked whether data were available in a way that enabled reanalysis.

RESULTS

Of 3556 analyzed articles, 3416 contained the DAS. The most frequent DAS category (42%) indicated that the data sets are available on reasonable request. Among 1792 manuscripts in which the DAS indicated that authors are willing to share their data, 1669 (93%) authors either did not respond or declined to share their data with us. Among 254 (14%) of 1792 authors who responded to our query for data sharing, only 123 (6.8%) provided the requested data.

CONCLUSION

Even when authors indicate in their manuscript that they will share data upon request, the compliance rate is the same as for authors who do not provide the DAS, suggesting that the DAS may not be sufficient to ensure data sharing.

Radiotheranostics in oncology: current challenges and emerging opportunities

Structural imaging remains an essential component of diagnosis, staging and response assessment in patients with cancer; however, as clinicians increasingly seek to noninvasively investigate tumour phenotypes and evaluate functional and molecular responses to therapy, theranostics - the combination of diagnostic imaging with targeted therapy - is becoming more widely implemented. The field of radiotheranostics, which is the focus of this Review, combines molecular imaging (primarily PET and SPECT) with targeted radionuclide therapy, which involves the use of small molecules, peptides and/or antibodies as carriers for therapeutic radionuclides, typically those emitting α-, β- or auger-radiation. The exponential, global expansion of radiotheranostics in oncology stems from its potential to target and eliminate tumour cells with minimal adverse effects, owing to a mechanism of action that differs distinctly from that of most other systemic therapies. Currently, an enormous opportunity exists to expand the number of patients who can benefit from this technology, to address the urgent needs of many thousands of patients across the world. In this Review, we describe the clinical experience with established radiotheranostics as well as novel areas of research and various barriers to progress.

Data-sharing and re-analysis for main studies assessed by the European Medicines Agency-a cross-sectional study on European Public Assessment Reports

BACKGROUND

Transparency and reproducibility are expected to be normative practices in clinical trials used for decision-making on marketing authorisations for new medicines. This registered report introduces a cross-sectional study aiming to assess inferential reproducibility for main trials assessed by the European Medicines Agency.

METHODS

Two researchers independently identified all studies on new medicines, biosimilars and orphan medicines given approval by the European Commission between January 2017 and December 2019, categorised as 'main studies' in the European Public Assessment Reports (EPARs). Sixty-two of these studies were randomly sampled. One researcher retrieved the individual patient data (IPD) for these studies and prepared a dossier for each study, containing the IPD, the protocol and information on the conduct of the study. A second researcher who had no access to study reports used the dossier to run an independent re-analysis of each trial. All results of these re-analyses were reported in terms of each study's conclusions, p-values, effect sizes and changes from the initial protocol. A team of two researchers not involved in the re-analysis compared results of the re-analyses with published results of the trial.

RESULTS

Two hundred ninety-two main studies in 173 EPARs were identified. Among the 62 studies randomly sampled, we received IPD for 10 trials. The median number of days between data request and data receipt was 253 [interquartile range 182-469]. For these ten trials, we identified 23 distinct primary outcomes for which the conclusions were reproduced in all re-analyses. Therefore, 10/62 trials (16% [95% confidence interval 8% to 28%]) were reproduced, as the 52 studies without available data were considered non-reproducible. There was no change from the original study protocol regarding the primary outcome in any of these ten studies. Spin was observed in the report of one study.

CONCLUSIONS

Despite their results supporting decisions that affect millions of people's health across the European Union, most main studies used in EPARs lack transparency and their results are not reproducible for external researchers. Re-analyses of the few trials with available data showed very good inferential reproducibility.

TRIAL REGISTRATION

https://osf.io/mcw3t/.

Beneficial effects of pumpkin seed soft extract on lower urinary tract symptoms and quality of life in men with benign prostatic hyperplasia: a meta-analysis of two randomized, placebo-controlled trials over 12 months

Background

In clinical practice, plant extracts are an option to treat mild-to-moderate lower urinary tract symptoms suggestive of benign prostate hyperplasia (LUTS/BPH). However, only a few herbal extracts have been investigated in long-term placebo-controlled studies. The safety and efficacy of a well-tolerated proprietary pumpkin seed soft extract (PSE) were investigated in two randomized placebo-controlled 12-month studies (Bach and GRANU study). Both trials studied LUTS/BPH patients with an International Prostate Symptom Score (IPSS) ≥13 points at baseline. The Bach study demonstrated positive effects of PSE compared to placebo, but no difference between treatments was observed in the GRANU study. We aimed to assess the efficacy of PSE in a meta-analysis using the patient-level data of these two studies.

Methods

Pooled analysis was performed in the intention-to-treat set using last-observation-carried-forward (ITT-LOCF). An IPSS improvement of ≥5 points after 12 months of therapy was the predefined response criterion. Logistic regression and ANCOVA models included the covariables treatment group, study, center size, and baseline IPSS. Each analysis was repeated for the per-protocol (PP) set.

Results

The ITT/PP analysis sets consisted of 687/485 and 702/488 patients in the PSE and placebo groups, respectively. At the 12-month follow-up, the response rates in the PSE group were 3% (ITT) and 5% (PP) higher than those in the placebo group. The odds ratio of response obtained by logistic regression analysis for comparing PSE versus placebo was 1.2 (95% CI 0.9, 1.5), favoring PSE (ITT- LOCF). For the IPSS change from baseline to 12 months, the ANCOVA estimated difference between the treatment groups was 0.7 points (95% CI 0.1, 1.2) in favor of PSE. The variables study, baseline IPSS, and center size had a relevant influence on treatment response.

Conclusion

Although the Bach and the GRANU study showed contradictory results, the analysis in a pooled form still pointed towards an advantage of PSE; namely, more patients in the PSE group showed an IPSS improvement of at least 5 points after 12 months. Therefore, the results of this meta-analysis suggest that patients with moderate LUTS/BPH may benefit from PSE treatment in terms of symptomatic relief.

Assessment of transparency and selective reporting of interventional trials studying colorectal cancer

Background

Colorectal cancer (CRC) is currently one of the most frequently diagnosed cancers. Our aim was to evaluate transparency and selective reporting in interventional trials studying CRC.

Methods

First, we assessed indicators of transparency with completeness of reporting, according to the CONSORT statement, and data sharing. We evaluated a selection of reporting items for a sample of randomized controlled trials (RCTs) studying CRC with published full-text articles between 2021–03-22 and 2018–03-22. Selected items were issued from the previously published CONSORT based peer-review tool (COBPeer tool). Then, we evaluated selective reporting through retrospective registration and primary outcome(s) switching between registration and publication. Finally, we determined if primary outcome(s) switching favored significant outcomes.

Results

We evaluated 101 RCTs with published full-text articles between 2021–03-22 and 2018–03-22. Five trials (5%) reported all selected CONSORT items completely. Seventy-four (73%), 53 (52%) and 13 (13%) trials reported the primary outcome(s), the allocation concealment process and harms completely. Twenty-five (25%) trials were willing to share data. In our sample, 49 (49%) trials were retrospectively registered and 23 (23%) trials had primary outcome(s) switching. The influence of primary outcome(s) switching could be evaluated in 16 (16/23 = 70%) trials, with 6 (6/16 = 38%) trials showing a discrepancy that favored statistically significant results.

Conclusions

Our results highlight a lack of transparency as well as frequent selective reporting in interventional trials studying CRC.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12885-022-09334-5.

COVID-19: A Catalyst for Transforming Randomized Trials

The coronavirus disease 2019 (COVID-19) pandemic incited a global clinical trial research agenda of unprecedented speed and high volume. This expedited research activity in a time of crisis produced both successes and failures that offer valuable learning opportunities for the scientific community to consider. Successes include the implementation of large adaptive and pragmatic trials as well as burgeoning efforts toward rapid data synthesis and open science principles. Conversely, notable failures include: (1) inadequate study design and execution; (2) data reversal, fraud, and retraction; and (3) research duplication and waste. Other challenges that became highlighted were the need to find unbiased designs for investigating complex, nonpharmaceutical interventions and the use of routinely collected data for outcomes assessment. This article discusses these issues juxtaposing the COVID-19 trials experience against trials in anesthesiology and other fields. These lessons may serve as a positive catalyst for transforming future clinical trial research.

Clinical pharmacology: Current innovations and future challenges

Clinical pharmacology is the study of drugs in humans, from first-in-human studies to randomized controlled trials (RCTs) and benefit-risk ratio assessment in large populations. The objective of this review is to present the recent innovations that may revolutionize the development of drugs in the future. On behalf of the French Society of Pharmacology and Therapeutics, we provide recommendations to address those future challenges in clinical pharmacology. Whatever the future will be, robust preliminary data on drug mechanism of action and rigorous study design will remain crucial prior to the start of pharmacological studies in human. At the present time, RCTs remains the gold standard to evaluate the efficacy of human drugs, although alternative designs (pragmatic trials, platform trials, etc.) are emerging. Innovations in healthy volunteers' studies and the contribution of new technologies such as artificial intelligence, machine learning and internet-based trials have the potential to improve drug development. In the field of precision medicine, new disease phenotypes and endotypes will probably help to identify new pharmacological targets, responders to therapies and patients at risk for drug adverse events. In such a moving landscape, the development of translational research through academic and private partnership, transparent sharing of clinical trial data and enhanced interactions between drug experts, patients and the general public are priority areas for action.

An urgent call to raise the bar in oncology

Important breakthroughs in medical treatments have improved outcomes for patients suffering from several types of cancer. However, many oncological treatments approved by regulatory agencies are of low value and do not contribute significantly to cancer mortality reduction, but lead to unrealistic patient expectations and push even affluent societies to unsustainable health care costs. Several factors that contribute to approvals of low-value oncology treatments are addressed, including issues with clinical trials, bias in reporting, regulatory agency shortcomings and drug pricing. With the COVID-19 pandemic enforcing the elimination of low-value interventions in all fields of medicine, efforts should urgently be made by all involved in cancer care to select only high-value and sustainable interventions. Transformation of medical education, improvement in clinical trial design, quality, conduct and reporting, strict adherence to scientific norms by regulatory agencies and use of value-based scales can all contribute to raising the bar for oncology drug approvals and influence drug pricing and availability.

Applying artificial intelligence for cancer immunotherapy

Artificial intelligence (AI) is a general term that refers to the use of a machine to imitate intelligent behavior for performing complex tasks with minimal human intervention, such as machine learning; this technology is revolutionizing and reshaping medicine. AI has considerable potential to perfect health-care systems in areas such as diagnostics, risk analysis, health information administration, lifestyle supervision, and virtual health assistance. In terms of immunotherapy, AI has been applied to the prediction of immunotherapy responses based on immune signatures, medical imaging and histological analysis. These features could also be highly useful in the management of cancer immunotherapy given their ever-increasing performance in improving diagnostic accuracy, optimizing treatment planning, predicting outcomes of care and reducing human resource costs. In this review, we present the details of AI and the current progression and state of the art in employing AI for cancer immunotherapy. Furthermore, we discuss the challenges, opportunities and corresponding strategies in applying the technology for widespread clinical deployment. Finally, we summarize the impact of AI on cancer immunotherapy and provide our perspectives about underlying applications of AI in the future.

Status, use and impact of sharing individual participant data from clinical trials: a scoping review

OBJECTIVES

To explore the impact of data-sharing initiatives on the intent to share data, on actual data sharing, on the use of shared data and on research output and impact of shared data.

ELIGIBILITY CRITERIA

All studies investigating data-sharing practices for individual participant data (IPD) from clinical trials.

SOURCES OF EVIDENCE

We searched the Medline database, the Cochrane Library, the Science Citation Index Expanded and the Social Sciences Citation Index via Web of Science, and preprints and proceedings of the International Congress on Peer Review and Scientific Publication. In addition, we inspected major clinical trial data-sharing platforms, contacted major journals/publishers, editorial groups and some funders.

CHARTING METHODS

Two reviewers independently extracted information on methods and results from resources identified using a standardised questionnaire. A map of the extracted data was constructed and accompanied by a narrative summary for each outcome domain.

RESULTS

93 studies identified in the literature search (published between 2001 and 2020, median: 2018) and 5 from additional information sources were included in the scoping review. Most studies were descriptive and focused on early phases of the data-sharing process. While the willingness to share IPD from clinical trials is extremely high, actual data-sharing rates are suboptimal. A survey of journal data suggests poor to moderate enforcement of the policies by publishers. Metrics provided by platforms suggest that a large majority of data remains unrequested. When requested, the purpose of the reuse is more often secondary analyses and meta-analyses, rarely re-analyses. Finally, studies focused on the real impact of data-sharing were rare and used surrogates such as citation metrics.

CONCLUSIONS

There is currently a gap in the evidence base for the impact of IPD sharing, which entails uncertainties in the implementation of current data-sharing policies. High level evidence is needed to assess whether the value of medical research increases with data-sharing practices.

PRIME-IPD SERIES Part 1. The PRIME-IPD tool promoted verification and standardization of study datasets retrieved for IPD meta-analysis

OBJECTIVES

We describe a systematic approach to preparing data in the conduct of Individual Participant Data (IPD) analysis.

STUDY DESIGN AND SETTING

A guidance paper proposing methods for preparing individual participant data for meta-analysis from multiple study sources, developed by consultation of relevant guidance and experts in IPD. We present an example of how these steps were applied in checking data for our own IPD meta analysis (IPD-MA).

RESULTS

We propose five steps of Processing, Replication, Imputation, Merging, and Evaluation to prepare individual participant data for meta-analysis (PRIME-IPD). Using our own IPD-MA as an exemplar, we found that this approach identified missing variables and potential inconsistencies in the data, facilitated the standardization of indicators across studies, confirmed that the correct data were received from investigators, and resulted in a single, verified dataset for IPD-MA.

CONCLUSION

The PRIME-IPD approach can assist researchers to systematically prepare, manage and conduct important quality checks on IPD from multiple studies for meta-analyses. Further testing of this framework in IPD-MA would be useful to refine these steps.

Evidence-Based, Patient-Centered Treatment of Erythema Migrans in the United States

Lyme disease, often characterized as a readily treatable infection, can be a debilitating and expensive illness, especially when patients remain symptomatic following therapy for early disease. Identifying and promoting highly effective therapeutic interventions for US patients with erythema migrans (EM) rashes that return them to their pre-infection health status should be a priority. The recently released treatment recommendations by the Infectious Diseases Society of America/American Academy of Neurology/American College of Rheumatology (IDSA/AAN/ACR) for the treatment of US patients fall short of that goal. This paper reviews the US trial evidence regarding EM rashes, discusses the shortcomings of the IDSA/AAN/ACR recommendations in light of that evidence and offers evidence-based, patient-centered strategies for managing patients with erythema migrans lesions.

Social media attention and citations of published outputs from re-use of clinical trial data: a matched comparison with articles published in the same journals

BACKGROUND

Data-sharing policies in randomized clinical trials (RCTs) should have an evaluation component. The main objective of this case-control study was to assess the impact of published re-uses of RCT data in terms of media attention (Altmetric) and citation rates.

METHODS

Re-uses of RCT data published up to December 2019 (cases) were searched for by two reviewers on 3 repositories (CSDR, YODA project, and Vivli) and matched to control papers published in the same journal. The Altmetric Attention Score (primary outcome), components of this score (e.g. mention of policy sources, media attention) and the total number of citations were compared between these two groups.

RESULTS

89 re-uses were identified: 48 (53.9%) secondary analyses, 34 (38.2%) meta-analyses, 4 (4.5%) methodological analyses and 3 (3.4%) re-analyses. The median (interquartile range) Altmetric Attention Scores were 5.9 (1.3-22.2) for re-use and 2.8 (0.3-12.3) for controls (p = 0.14). No statistical difference was found on any of the components of in the Altmetric Attention Score. The median (interquartile range) numbers of citations were 3 (1-8) for reuses and 4 (1 - 11.5) for controls (p = 0.30). Only 6/89 re-uses (6.7%) were cited in a policy source.

CONCLUSIONS

Using all available re-uses of RCT data to date from major data repositories, we were not able to demonstrate that re-uses attracted more attention than a matched sample of studies published in the same journals. Small average differences are still possible, as the sample size was limited. However matching choices have some limitations so results should be interpreted very cautiously. Also, citations by policy sources for re-uses were rare.

TRIAL REGISTRATION

Registration: osf.io/fp62e.

Intent to share Annals of Internal Medicine's trial data was not associated with data re-use

OBJECTIVE

To explore the impact of the Annals of Internal Medicine (AIM) data-sharing policy for randomized controlled trials (RCTs) in terms of output from data-sharing (i.e. publications re-using the data).

STUDY DESIGN AND SETTING

Retrospective study. RCTs published in the AIM between 2007 and 2017 were retrieved on PubMed. Publications where the data had been re-used were identified on Web of Science. Searches were performed by two independent reviewers. The primary outcome was any published re-use of the data (re-analysis, secondary analysis, or meta-analysis of individual participant data [MIPD]), where the first, last and corresponding authors were not among the authors of the RCT. Analyses used Cox (primary analysis) models adjusting for RCTs characteristics (registration: https://osf.io/8pj5e/).

RESULTS

185 RCTs were identified. 106 (57%) mentioned willingness to share data and 79 (43%) did not. 208 secondary analyses, 67 MIPD and no re-analyses were identified. No significant association was found between intent to share and re-use where the first, last and corresponding authors were not among the authors of the primary RCT (adjusted hazard ratio = 1.04 [0.47-2.30]).

CONCLUSION

Over ten years, RCTs published in AIM expressing an intention to share data were not associated with more extensive re-use of the data.

Can synthetic data be a proxy for real clinical trial data? A validation study

OBJECTIVES

There are increasing requirements to make research data, especially clinical trial data, more broadly available for secondary analyses. However, data availability remains a challenge due to complex privacy requirements. This challenge can potentially be addressed using synthetic data.

SETTING

Replication of a published stage III colon cancer trial secondary analysis using synthetic data generated by a machine learning method.

PARTICIPANTS

There were 1543 patients in the control arm that were included in our analysis.

PRIMARY AND SECONDARY OUTCOME MEASURES

Analyses from a study published on the real dataset were replicated on synthetic data to investigate the relationship between bowel obstruction and event-free survival. Information theoretic metrics were used to compare the univariate distributions between real and synthetic data. Percentage CI overlap was used to assess the similarity in the size of the bivariate relationships, and similarly for the multivariate Cox models derived from the two datasets.

RESULTS

Analysis results were similar between the real and synthetic datasets. The univariate distributions were within 1% of difference on an information theoretic metric. All of the bivariate relationships had CI overlap on the tau statistic above 50%. The main conclusion from the published study, that lack of bowel obstruction has a strong impact on survival, was replicated directionally and the HR CI overlap between the real and synthetic data was 61% for overall survival (real data: HR 1.56, 95% CI 1.11 to 2.2; synthetic data: HR 2.03, 95% CI 1.44 to 2.87) and 86% for disease-free survival (real data: HR 1.51, 95% CI 1.18 to 1.95; synthetic data: HR 1.63, 95% CI 1.26 to 2.1).

CONCLUSIONS

The high concordance between the analytical results and conclusions from synthetic and real data suggests that synthetic data can be used as a reasonable proxy for real clinical trial datasets.

TRIAL REGISTRATION NUMBER

NCT00079274.

Misinformation in and about science

Humans learn about the world by collectively acquiring information, filtering it, and sharing what we know. Misinformation undermines this process. The repercussions are extensive. Without reliable and accurate sources of information, we cannot hope to halt climate change, make reasoned democratic decisions, or control a global pandemic. Most analyses of misinformation focus on popular and social media, but the scientific enterprise faces a parallel set of problems-from hype and hyperbole to publication bias and citation misdirection, predatory publishing, and filter bubbles. In this perspective, we highlight these parallels and discuss future research directions and interventions.

Optimizing the synthesis of clinical trial data using sequential trees

OBJECTIVE

With the growing demand for sharing clinical trial data, scalable methods to enable privacy protective access to high-utility data are needed. Data synthesis is one such method. Sequential trees are commonly used to synthesize health data. It is hypothesized that the utility of the generated data is dependent on the variable order. No assessments of the impact of variable order on synthesized clinical trial data have been performed thus far. Through simulation, we aim to evaluate the variability in the utility of synthetic clinical trial data as variable order is randomly shuffled and implement an optimization algorithm to find a good order if variability is too high.

MATERIALS AND METHODS

Six oncology clinical trial datasets were evaluated in a simulation. Three utility metrics were computed comparing real and synthetic data: univariate similarity, similarity in multivariate prediction accuracy, and a distinguishability metric. Particle swarm was implemented to optimize variable order, and was compared with a curriculum learning approach to ordering variables.

RESULTS

As the number of variables in a clinical trial dataset increases, there is a pattern of a marked increase in variability of data utility with order. Particle swarm with a distinguishability hinge loss ensured adequate utility across all 6 datasets. The hinge threshold was selected to avoid overfitting which can create a privacy problem. This was superior to curriculum learning in terms of utility.

CONCLUSIONS

The optimization approach presented in this study gives a reliable way to synthesize high-utility clinical trial datasets.

Randomised controlled trials for COVID-19: evaluation of optimal randomisation methodologies-need for data validation of the completed trials and to improve ongoing and future randomised trial designs

During the emerging COVID-19 (coronavirus disease 2019) pandemic, initially there were no proven treatment options. With the release of randomised controlled trial (RCT) results, we are beginning to see possible treatment options for COVID-19. The RECOVERY trial showed an absolute risk reduction in mortality by 2.8% with dexamethasone, and the ACTT-1 trial showed that treatment with remdesivir reduced the time to recovery by 4 days. Treatment with hydroxychloroquine (HCQ) and lopinavir/ritonavir did not show any mortality benefit in either the RECOVERY or World Health Organization (WHO) Solidarity trials. The National Institutes of Health (NIH) and Brazilian HCQ trials did not show any benefit for HCQ based on the seven-point ordinal scale outcomes. The randomisation methodologies utilised in these controlled trials and the quality of published data were reviewed to examine their adaptability to treat patients. We found that the randomisation methodologies of these trials were suboptimal for matching the studied groups based on disease severity among critically-ill hospitalised COVID-19 patients with high mortality rates. The published literature is very limited regarding the disease severity metrics among the compared groups and failed to show that the data are without fatal sampling errors and sampling biases. We also found that there is a definite need for the validation of data in these trials along with additional important disease severity metrics to ensure that the trials' conclusions are accurate. We also propose proper randomisation methodologies for the design of RCTs for COVID-19 as well as guidance for the publication of COVID-19 trial results.

Data management in substance use disorder treatment research: Implications from data harmonization of National Institute on Drug Abuse-funded randomized controlled trials

Background:

Secondary analysis of data from completed randomized controlled trials is a critical and efficient way to maximize the potential benefits from past research. De-identified primary data from completed randomized controlled trials have been increasingly available in recent years; however, the lack of standardized data products is a major barrier to further use of these valuable data. Pre-statistical harmonization of data structure, variables, and codebooks across randomized controlled trials would facilitate secondary data analysis, including meta-analyses and comparative effectiveness studies. We describe a pre-statistical data harmonization initiative to standardize de-identified primary data from substance use disorder treatment randomized controlled trials funded by the National Institute on Drug Abuse available on the National Institute on Drug Abuse Data Share website.

Methods:

Standardized datasets and codebooks with consistent data structures, variable names, labels, and definitions were developed for 36 completed randomized controlled trials. Common data domains were identified to bundle data files from individual randomized controlled trials according to relevant concepts. Variables were harmonized if at least two randomized controlled trials used the same instruments. The structures of the harmonized data were determined based on the feedback from clinical trialists and substance use disorder research experts.

Results:

We have created a harmonized database of variables across 36 randomized controlled trials with a build-in label and a brief definition for each variable. Data files from the randomized controlled trials have been consistently categorized into eight domains (enrollment, demographics, adherence, adverse events, physical health measures, mental-behavioral-cognitive health measures, self-reported substance use measures, and biologic substance use measures). Standardized codebooks and concordance tables have also been developed to help identify instruments and variables of interest more easily.

Conclusion:

The harmonized data of randomized controlled trials of substance use disorder treatments can potentially promote future secondary data analysis of completed randomized controlled trials, allowing combining data from multiple randomized controlled trials and provide guidance for future randomized controlled trials in substance use disorder treatment research.

Meta-analyses of diagnostic test accuracy could not be reproduced

BACKGROUND AND OBJECTIVES

The aim of our study was to investigate the reproducibility of diagnostic accuracy meta-analyses, as reported in published systematic reviews.

STUDY DESIGN AND SETTING

We selected all systematic reviews of diagnostic test accuracy containing a meta-analysis, published in January 2018 and retrieved in Medline through Ovid. All reviews reported a summary estimate of sensitivity and specificity. We requested the protocol from their authors and used the protocol and the information in the published review to reproduce the reported meta-analysis. Successful reproduction was defined as a result differing <1% point from the reported point estimates; or reported primary study results that were in line with those of the actual primary study results; or if the data from the primary studies could be extracted without checking the data in the review first.

RESULTS

Of the 51 included reviews, 16 had a protocol registered in PROSPERO and five of those responded to our request for a protocol. Nineteen reviews (37%) provided the 2×2 tables that were included in the meta-analysis. In 14 of those, the outcome of the meta-analysis could be reproduced. Considering the correctness of the numbers from the primary articles and the complete reporting of the search strategy, only one meta-analysis was fully replicable.

CONCLUSION

Published meta-analyses of diagnostic test accuracy were poorly replicable. This was partly because of lack of information about the methods and data used, and partly because of mistakes in the data extraction or data reporting.

Changes to aspects of ongoing randomised controlled trials with fixed designs

Background

Despite careful planning, changes to some aspects of an ongoing randomised clinical trial (RCT), with a fixed design, may be warranted. We sought to elucidate the distinction between legitimate versus illegitimate changes to serve as a guide for less experienced clinical trialists and other stakeholders.

Methods

Using data from a large trial of statin therapy for secondary prevention, we generated a set of simulated trial datasets under the null hypothesis (H0) and a set under an alternative hypothesis (H1). Through analysis of these simulated trials, we assessed the performance of the strategy of changing aspects of the design/analysis with knowledge of treatment allocation (illegitimate) versus the strategy of making changes without knowledge of treatment allocation (legitimate). Performance was assessed using the type 1 error, as well as measures of absolute and relative bias in the treatment effect.

Results

Illegitimate changes led to a relative bias of 61% under H1, and a type 1 error rate under H0 of 23%—well in excess of the 5% significance level targeted. Legitimate changes produced unbiased estimates under H1 and did not inflate the type 1 error rate under H0.

Conclusions

Changes to pre-specified aspects of the design and analysis of an ongoing RCT may be a necessary response to unforeseen circumstances. Such changes risk introducing a bias if undertaken with knowledge of treatment allocation. Legitimate changes need to be adequately documented to provide assurance to all stakeholders of their validity.

Novel trial strategies to enhance the relevance, efficiency, effectiveness, and impact of nephrology research

Randomized controlled trials (RCTs) are considered the gold standard for evaluating the effectiveness of interventions. However, criticisms of traditional designs are that they can be inefficient, inflexible, expensive, and conducted in a manner disconnected from real-life clinical practice. Novel strategies and approaches are being utilized to overcome these limitations, including comprehensive consumer engagement, core outcome sets, novel trial designs, streamlined data collection, cost-effectiveness and return on investment evaluations, knowledge dissemination plans, and impact evaluation. These strategies can be implemented at the design, conduct, implementation, and dissemination stages of the trial process. This review aims to provide an overview of these strategies and approaches to improve the relevance, efficiency, effectiveness, and impact of nephrology research.

Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies

OBJECTIVE

To systematically examine the design, reporting standards, risk of bias, and claims of studies comparing the performance of diagnostic deep learning algorithms for medical imaging with that of expert clinicians.

DESIGN

Systematic review.

DATA SOURCES

Medline, Embase, Cochrane Central Register of Controlled Trials, and the World Health Organization trial registry from 2010 to June 2019.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

Randomised trial registrations and non-randomised studies comparing the performance of a deep learning algorithm in medical imaging with a contemporary group of one or more expert clinicians. Medical imaging has seen a growing interest in deep learning research. The main distinguishing feature of convolutional neural networks (CNNs) in deep learning is that when CNNs are fed with raw data, they develop their own representations needed for pattern recognition. The algorithm learns for itself the features of an image that are important for classification rather than being told by humans which features to use. The selected studies aimed to use medical imaging for predicting absolute risk of existing disease or classification into diagnostic groups (eg, disease or non-disease). For example, raw chest radiographs tagged with a label such as pneumothorax or no pneumothorax and the CNN learning which pixel patterns suggest pneumothorax.

REVIEW METHODS

Adherence to reporting standards was assessed by using CONSORT (consolidated standards of reporting trials) for randomised studies and TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) for non-randomised studies. Risk of bias was assessed by using the Cochrane risk of bias tool for randomised studies and PROBAST (prediction model risk of bias assessment tool) for non-randomised studies.

RESULTS

Only 10 records were found for deep learning randomised clinical trials, two of which have been published (with low risk of bias, except for lack of blinding, and high adherence to reporting standards) and eight are ongoing. Of 81 non-randomised clinical trials identified, only nine were prospective and just six were tested in a real world clinical setting. The median number of experts in the comparator group was only four (interquartile range 2-9). Full access to all datasets and code was severely limited (unavailable in 95% and 93% of studies, respectively). The overall risk of bias was high in 58 of 81 studies and adherence to reporting standards was suboptimal (<50% adherence for 12 of 29 TRIPOD items). 61 of 81 studies stated in their abstract that performance of artificial intelligence was at least comparable to (or better than) that of clinicians. Only 31 of 81 studies (38%) stated that further prospective studies or trials were required.

CONCLUSIONS

Few prospective deep learning studies and randomised trials exist in medical imaging. Most non-randomised trials are not prospective, are at high risk of bias, and deviate from existing reporting standards. Data and code availability are lacking in most studies, and human comparator groups are often small. Future studies should diminish risk of bias, enhance real world clinical relevance, improve reporting and transparency, and appropriately temper conclusions.

STUDY REGISTRATION

PROSPERO CRD42019123605.

Evaluating the re-identification risk of a clinical study report anonymized under EMA Policy 0070 and Health Canada Regulations

BACKGROUND

Regulatory agencies, such as the European Medicines Agency and Health Canada, are requiring the public sharing of clinical trial reports that are used to make drug approval decisions. Both agencies have provided guidance for the quantitative anonymization of these clinical reports before they are shared. There is limited empirical information on the effectiveness of this approach in protecting patient privacy for clinical trial data.

METHODS

In this paper we empirically test the hypothesis that when these guidelines are implemented in practice, they provide adequate privacy protection to patients. An anonymized clinical study report for a trial on a non-steroidal anti-inflammatory drug that is sold as a prescription eye drop was subjected to re-identification. The target was 500 patients in the USA. Only suspected matches to real identities were reported.

RESULTS

Six suspected matches with low confidence scores were identified. Each suspected match took 24.2 h of effort. Social media and death records provided the most useful information for getting the suspected matches.

CONCLUSIONS

These results suggest that the anonymization guidance from these agencies can provide adequate privacy protection for patients, and the modes of attack can inform further refinements of the methodologies they recommend in their guidance for manufacturers.

DEBATE-statistical analysis plans for observational studies

Background

All clinical research benefits from transparency and validity. Transparency and validity of studies may increase by prospective registration of protocols and by publication of statistical analysis plans (SAPs) before data have been accessed to discern data-driven analyses from pre-planned analyses.

Main message

Like clinical trials, recommendations for SAPs for observational studies increase the transparency and validity of findings. We appraised the applicability of recently developed guidelines for the content of SAPs for clinical trials to SAPs for observational studies. Of the 32 items recommended for a SAP for a clinical trial, 30 items (94%) were identically applicable to a SAP for our observational study. Power estimations and adjustments for multiplicity are equally important in observational studies and clinical trials as both types of studies usually address multiple hypotheses. Only two clinical trial items (6%) regarding issues of randomisation and definition of adherence to the intervention did not seem applicable to observational studies. We suggest to include one new item specifically applicable to observational studies to be addressed in a SAP, describing how adjustment for possible confounders will be handled in the analyses.

Conclusion

With only few amendments, the guidelines for SAP of a clinical trial can be applied to a SAP for an observational study. We suggest SAPs should be equally required for observational studies and clinical trials to increase their transparency and validity.

Role of industry funders in oncology RCTs published in high-impact journals and its association with trial conclusions and time to publication

Background

Previous studies have shown that industry funded trials are associated with pro-industry conclusions and publication bias. Less is known about the role of industry funders and their influence on trial conclusions and time to publication.

Methods

We identified all industry funded RCTs published in six high-impact clinical journals between 2014 and 2016 to estimate the prevalence of the role of industry funders in trial design, data collection, data analyses, data interpretation and manuscript writing. Ordinal logistic regression was used to assess the association between the role of industry funders and trial conclusions, which was classified on a five-point scale. Cox proportional-hazards were used to examine the effect of role of funder on time to publication.

Results

Of the 255 eligible RCTs, industry funders had a role in trial design in 179 (70.2%) trials, data collection in 160 (62.7%) trials, data analyses in 173 (67.8%) trials, data interpretation in 135 (52.9%) trials and manuscript writing in 168 (65.9%) trials. Trials with any role of industry funders had 3.6 times (95% CI 2.0-6.6) higher odds of having positive conclusions compared with those without role of industry funders. In trials with any role of industry funders, positive trials were published more rapidly than negative trials (hazard ratio = 4.3; 95% CI 2.7-6.7, P < 0.001), while for trials without role of industry funders, there was no association (hazard ratio = 1.07; 95% CI 0.57-1.99, P = 0.84).

Conclusion

The involvement of industry funders is common in all stages of clinical trials and was associated with more positive conclusions and more rapid publication of RCTs with positive results.

Increase value and reduce waste in research on psychological therapies

A seminal Lancet series focused on increasing value and reducing waste in biomedical research, providing a transferrable template to diagnose problems in research. Our goal was to document how some of these sources of waste apply to mental health and particularly psychological treatments research. We synthesize and critically evaluate empirical findings in relation to four major sources: i) defining research priorities; ii) research design, methods and analysis; iii) accessibility of research information; iv) accuracy and usability of research reports. We demonstrate that each source of waste considered is well-represented and amply documented within this field. We describe hype and insufficient consideration of what is known in defining research priorities, persistent risk of bias, particularly due to selective outcome reporting, for psychotherapy trials across mental disorders, intellectual and financial biases, direct and indirect evidence of publication bias, largely inexistent adoption of data sharing, issues of multiplicity and fragmentation of data and findings, and insufficient adoption of reporting guidelines. We expand on a few general solutions, including supporting meta-research, properly testing interventions to increase research quality, placing open science at the center of psychological treatment research and remaining vigilant particularly regarding the strains of research currently prioritized, such as experimental psychopathology.

Feasibility, Process, and Outcomes of Cardiovascular Clinical Trial Data Sharing: A Reproduction Analysis of the SMART-AF Trial

Importance

Data sharing is as an expanding initiative for enhancing trust in the clinical research enterprise.

Objective

To evaluate the feasibility, process, and outcomes of a reproduction analysis of the THERMOCOOL SMARTTOUCH Catheter for the Treatment of Symptomatic Paroxysmal Atrial Fibrillation (SMART-AF) trial using shared clinical trial data.

Design, Setting, and Participants

A reproduction analysis of the SMART-AF trial was performed using the data sets, data dictionary, case report file, and statistical analysis plan from the original trial accessed through the Yale Open Data Access Project using the SAS Clinical Trials Data Transparency platform. SMART-AF was a multicenter, single-arm trial evaluating the effectiveness and safety of an irrigated, contact force-sensing catheter for ablation of drug refractory, symptomatic paroxysmal atrial fibrillation in 172 participants recruited from 21 sites between June 2011 and December 2011. Analysis of the data was conducted between December 2016 and April 2017.

Main Outcomes and Measures

Effectiveness outcomes included freedom from atrial arrhythmias after ablation and proportion of participants without any arrhythmia recurrence over the 12 months of follow-up after a 3-month blanking period. Safety outcomes included major adverse device- or procedure-related events.

Results

The SMART AF trial participants' mean age was 58.7 (10.8) years, and 72% were men. The time from initial proposal submission to final analysis was 11 months. Freedom from atrial arrhythmias at 12 months postprocedure was similar compared with the primary study report (74.0%; 95% CI, 66.0-82.0 vs 76.4%; 95% CI, 68.7-84.1). The reproduction analysis success rate was higher than the primary study report (65.8%; 95% CI 56.5-74.2 vs 75.6%; 95% CI, 67.2-82.5). Adverse events were minimal and similar between the 2 analyses, but contact force range or regression models could not be reproduced.

Conclusions and Relevance

The feasibility of a reproduction analysis of the SMART-AF trial was demonstrated through an academic data-sharing platform. Data sharing can be facilitated through incentivizing collaboration, sharing statistical code, and creating more decentralized data sharing platforms with fewer restrictions to data access.

Verify original results through reanalysis before replicating

In determining the need to directly replicate, it is crucial to first verify the original results through independent reanalysis of the data. Original results that appear erroneous and that cannot be reproduced by reanalysis offer little evidence to begin with, thereby diminishing the need to replicate. Sharing data and scripts is essential to ensure reproducibility.

A Future for Observational Epidemiology: Clarity, Credibility, Transparency

Observational studies are ambiguous, difficult, and necessary for epidemiology. Presently, there are concerns that the evidence produced by most observational studies in epidemiology is not credible and contributes to research waste. I argue that observational epidemiology could be improved by focusing greater attention on 1) defining questions that make clear whether the inferential goal is descriptive or causal; 2) greater utilization of quantitative bias analysis and alternative research designs that aim to decrease the strength of assumptions needed to estimate causal effects; and 3) promoting, experimenting with, and perhaps institutionalizing both reproducible research standards and replication studies to evaluate the fragility of study findings in epidemiology. Greater clarity, credibility, and transparency in observational epidemiology will help to provide reliable evidence that can serve as a basis for making decisions about clinical or population-health interventions.

Reproducible pharmacokinetics

Reproducibility is a highly desired feature of scientific investigation in general, and it has special connotations for research in pharmacokinetics, a vibrant field with over 500,000 publications to-date. It is important to be able to differentiate between genuine heterogeneity in pharmacokinetic parameters from heterogeneity that is due to errors and biases. This overview discusses efforts and opportunities to diminish the latter type of undesirable heterogeneity. Several reporting and research guidance documents and standards have been proposed for pharmacokinetic studies, but their adoption is still rather limited. Quality problems in the methods used and model evaluations have been examined in some empirical studies of the literature. Standardization of statistical and laboratory tools and procedures can be improved in the field. Only a small fraction of pharmacokinetic studies become pre-registered and only 9995 such studies have been registered in ClinicalTrials.gov as of August 2018. It is likely that most pharmacokinetic studies remain unpublished. Publication bias affecting the results and inferences has been documented in case studies, but its exact extent is unknown for the field at-large. The use of meta-analyses in the field is still limited. Availability of raw data, detailed protocols, software and codes is hopefully improving with multiple ongoing initiatives. Several research practices can contribute to greater transparency and reproducibility for pharmacokinetic investigations.

The role of meta-analyses and umbrella reviews in assessing the harms of psychotropic medications: beyond qualitative synthesis

ὠφελέειν, ἢ μὴ βλάπτειν (Primum non nocere) - Hιppocrates' principle should still guide daily medical prescribing. Therefore, assessing evidence of psychopharmacologic agents' safety and harms is essential. Randomised controlled trials (RCTs) and observational studies may provide complementary information about harms of psychopharmacologic medications from both experimental and real-world settings. It is considered that RCTs provide a better control of confounding variables, while observational studies provide evidence from larger samples, longer follow-ups, in more representative samples, which may be more reflective of real-life clinical scenarios. However, this may not always hold true. Moreover, in observational studies, safety data are poorly or inconsistently reported, precluding reliable quantitative synthesis in meta-analyses. Beyond individual studies, meta-analyses, which represent the highest level of 'evidence', can be misleading, redundant and of low methodological quality. Overlapping meta-analyses sometimes even reach different conclusions on the same topic. Meta-analyses should be assessed systematically. Descriptive reviews of reviews can be poorly informative. Conversely, 'umbrella reviews' can use a quantitative approach to grade evidence. In this editorial, we present the main factors involved in the assessment of psychopharmacologic agents' harms from individual studies, meta-analyses and umbrella reviews. Study design features, sample size, number of the events of interest, summary effect sizes, p-values, heterogeneity, 95% prediction intervals, confounding factor adjustment and tests of bias (e.g., small-study effects and excess significance) can be combined with other assessment tools, such as AMSTAR and GRADE to create a framework for assessing the credibility of evidence.

Enhancing patient-level clinical data access to promote evidence-based practice and incentivize therapeutic innovation

Clinical trials are crucial to determining the human safety and efficacy of new therapeutic innovations. Extraordinary amounts of human experiential data are generated over the course of any clinical trial, however, much of these data is never made publicly accessible. Improved, reliable data sharing is essential to inform clinical decisions and incentivize further therapeutic improvements; this need, and the call and concept to enhance patient-level clinical trial data accessibility is not new. Several recent public and private shifts in clinical data sharing policies and procedures promise to improve access and data utility to reduce waste in research and increase efficiency of evidence synthesis. Nonetheless, pharmaceutical industry remain reluctant to share full clinical data sets at some level to protect their commercial interests and avoid misuse of their data. Here, we review the landscape of emerging regulations related to the sharing of patient level data and current clinical data access models of major pharmaceutical companies. We also summarize the different measures that could satisfy both clinical data producers and users in achieving the benefits of accessing patient-level data while mitigating any associated risks.

Variation in analytic transparency in recent efficacy studies of antidepressant medication

The validity of clinical trial results is influenced by researchers' decisions regarding the management of missing data. Inadequate management of missing data has been identified as a significant source of bias that can result in an overestimation of drug efficacy. Transparency related to the management of missing data is essential to assess the strength of evidence reported in publications. In a subset of 17 randomised clinical trials for two new antidepressant medications, we present a case study in which we examined investigators' decisions regarding how to handle missing data and if their chosen method took into account, possible violations of analytic requirements that could affect results. The majority of trials (76%) concluded that there was a benefit of antidepressant treatment and in 94% the methodology for handling missing data was identifiable. Of these, 50% imputed data using the last observation carried forward and half used a mixed-effects model repeated measure approach. Most reports did not provide a rationale for the method used, and no trials described analyses regarding differences between completers and dropouts. Sensitivity analysis was inconsistently reported and correction for multiple comparisons was not uniformly applied. Lack of transparency for analytic choices related to handling of missing data testing was common in this subset of RCTs. Because management of missing data can directly influence the quality of study results, it is critical that journal editors develop and enforce standards for methodological transparency.

Caveat emptor: the combined effects of multiplicity and selective reporting

Clinical trials and systematic reviews of clinical trials inform healthcare decisions. There is growing concern, however, about results from clinical trials that cannot be reproduced. Reasons for nonreproducibility include that outcomes are defined in multiple ways, results can be obtained using multiple methods of analysis, and trial findings are reported in multiple sources ("multiplicity"). Multiplicity combined with selective reporting can influence dissemination of trial findings and decision-making. In particular, users of evidence might be misled by exposure to selected sources and overly optimistic representations of intervention effects. In this commentary, drawing from our experience in the Multiple Data Sources in Systematic Reviews (MUDS) study and evidence from previous research, we offer practical recommendations to enhance the reproducibility of clinical trials and systematic reviews.