Reducing waste from incomplete or unusable reports of biomedical research

Recruitment, consent and retention of participants in randomised controlled trials: a review of trials published in the National Institute for Health Research (NIHR) Journals Library (1997-2020)

OBJECTIVES

To review the consent, recruitment and retention rates for randomised controlled trials (RCTs) funded by the UK's National Institute for Health Research (NIHR) and published in the online NIHR Journals Library between January 1997 and December 2020.

DESIGN

Comprehensive review.

SETTING

RCTs funded by the NIHR and published in the NIHR Journals Library.

DATA EXTRACTION

Information relating to the trial characteristics, sample size, recruitment and retention.

PRIMARY AND SECONDARY OUTCOME MEASURES

The primary outcome was the recruitment rate (number of participants recruited per centre per month). Secondary outcomes were the target sample size and whether it was achieved; consent rates (percentage of eligible participants who consented and were randomised) and retention rates (percentage of randomised participants retained and assessed with valid primary outcome data).

RESULTS

This review identified 388 individual RCTs from 379 reports in the NIHR Journals Library. The final recruitment target sample size was achieved in 63% (245/388) of the RCTs. The original recruitment target was revised in 30% (118/388) of trials (downwards in 67% (79/118)). The median recruitment rate (participants per centre per month) was found to be 0.95 (IQR: 0.42-2.60); the median consent rate was 72% (IQR: 50%-88%) and the median retention rate was estimated at 88% (IQR: 80%-97%).

CONCLUSIONS

There is considerable variation in the consent, recruitment and retention rates in publicly funded RCTs. Although the majority of (6 out of 10) trials in this review achieved their final target sample; 3 out of 10 trials revised their original target sample size (downwards in 7 out of 10 trials). Investigators should bear this in mind at the planning stage of their study and not be overly optimistic about their recruitment projections.

Strengthening research capacity: a systematic review of manuscript writing and publishing interventions for researchers in low-income and middle-income countries

INTRODUCTION

Health researchers from low-income and middle-income countries (LMICs) are under-represented in the academic literature. Scientific writing and publishing interventions may help researchers publish their findings; however, we lack evidence about the prevalence and effectiveness of such interventions. This review describes interventions for researchers in LMICs aimed at strengthening capacity for writing and publishing academic journal articles.

METHODS

We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines to report literature searches in PubMed, Embase, Global Health, Scopus and ERIC. Our keywords included LMICs, low-income and middle-income countries, health research and writing/publication support interventions, with no restrictions on publication date. Our screening process consisted of title screening, abstract review and full-text review. We collected information about the content, implementation and evaluation of each intervention, if included.

RESULTS

We identified 20 interventions designed to strengthen capacity for scientific writing and publishing. We summarised information from the 14 interventions that reported submitted or published papers as outcomes separately, reasoning that because they provide quantifiable metrics of success, they may offer particular insights into intervention components leading to publication. The writing and publishing components in this 'Publications Reported' group were an average length of 5.4 days compared with 2.5 days in the other group we refer to as 'Other Interventions.' Whereas all 14 Publications Reported interventions incorporated mentors, only two of five in the Other Interventions group incorporated mentors. Across interventions, leaders expressed the importance of a high ratio of mentors to participants, the need to accommodate time demands of busy researchers, and the necessity of a budget for open access fees and high-quality internet connectivity.

CONCLUSION

Writing and publishing interventions in LMICs are an underutilised opportunity for capacity strengthening. To facilitate the implementation of high-quality interventions, future writing and publishing interventions should share their experiences by publishing detailed information about the approach and effectiveness of the interventions.

STAndard Reporting of CAries Detection and Diagnostic Studies (STARCARDDS)

AIM

The aim of this paper is to present recommendations from an international workshop which evaluated the methodology and reporting of caries diagnostic studies. As a unique feature, this type of studies is focused on caries lesion detection and assessment, and many of them are carried out in vitro, because of the possibility of histological validation of the whole caries spectrum. This feature is not well covered in the existing reporting STARD guideline within the EQUATOR Network.

PARTICIPANTS AND METHODS

An international working group of 13 cariology researchers was formed. The STARD checklist was reviewed and modified for caries detection and diagnosis purposes, in a three-step process of evaluation, consensual modification, and delivery during three 2-day workshops over 18 months. Special attention was paid to reporting requirements of caries studies that solely focus on reliability.

RESULTS

The STARD checklist was modified in 14/30 items, with an emphasis on issues of sample selection (tooth selection in in vitro studies), blinding, and detailed reporting of results.

CONCLUSION

Following STARCARDDS (STAndard Reporting of CAries Detection and Diagnostic Studies) is expected to result in complete reporting of study design and methodology in future caries diagnosis and detection experiments both in vivo and in vitro, thus allowing for better comparability of studies and higher quality of systematic reviews.

CLINICAL RELEVANCE

Standardization of caries diagnostic studies leads to a better comparability among future studies, both in vivo and in vitro.

How informative were early SARS-CoV-2 treatment and prevention trials? a longitudinal cohort analysis of trials registered on ClinicalTrials.gov

Background

Early in the SARS-CoV-2 pandemic, commentators warned that some COVID trials were inadequately conceived, designed and reported. Here, we retrospectively assess the prevalence of informative COVID trials launched in the first 6 months of the pandemic.

Methods

Based on prespecified eligibility criteria, we created a cohort of Phase 1/2, Phase 2, Phase 2/3 and Phase 3 SARS-CoV-2 treatment and prevention efficacy trials that were initiated from 2020-01-01 to 2020-06-30 using ClinicalTrials.gov registration records. We excluded trials evaluating behavioural interventions and natural products, which are not regulated by the U.S. Food and Drug Administration (FDA). We evaluated trials on 3 criteria of informativeness: potential redundancy (comparing trial phase, type, patient-participant characteristics, treatment regimen, comparator arms and primary outcome), trials design (according to the recommendations set-out in the May 2020 FDA guidance document on SARS-CoV-2 treatment and prevention trials) and feasibility of patient-participant recruitment (based on timeliness and success of recruitment).

Results

We included all 500 eligible trials in our cohort, 58% of which were Phase 2 and 84.8% were directed towards the treatment of SARS-CoV-2. Close to one third of trials met all three criteria and were deemed informative (29.9% (95% Confidence Interval 23.7–36.9)). The proportion of potentially redundant trials in our cohort was 4.1%. Over half of the trials in our cohort (56.2%) did not meet our criteria for high quality trial design. The proportion of trials with infeasible patient-participant recruitment was 22.6%.

Conclusions

Less than one third of COVID-19 trials registered on ClinicalTrials.gov during the first six months met all three criteria for informativeness. Shortcomings in trial design, recruitment feasibility and redundancy reflect longstanding weaknesses in the clinical research enterprise that were likely amplified by the exceptional circumstances of a pandemic.

Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review

BACKGROUND

While many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. We aim to systematically review the adherence of Machine Learning (ML)-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement.

METHODS

We included articles reporting on development or external validation of a multivariable prediction model (either diagnostic or prognostic) developed using supervised ML for individualized predictions across all medical fields. We searched PubMed from 1 January 2018 to 31 December 2019. Data extraction was performed using the 22-item checklist for reporting of prediction model studies ( www.TRIPOD-statement.org ). We measured the overall adherence per article and per TRIPOD item.

RESULTS

Our search identified 24,814 articles, of which 152 articles were included: 94 (61.8%) prognostic and 58 (38.2%) diagnostic prediction model studies. Overall, articles adhered to a median of 38.7% (IQR 31.0-46.4%) of TRIPOD items. No article fully adhered to complete reporting of the abstract and very few reported the flow of participants (3.9%, 95% CI 1.8 to 8.3), appropriate title (4.6%, 95% CI 2.2 to 9.2), blinding of predictors (4.6%, 95% CI 2.2 to 9.2), model specification (5.2%, 95% CI 2.4 to 10.8), and model's predictive performance (5.9%, 95% CI 3.1 to 10.9). There was often complete reporting of source of data (98.0%, 95% CI 94.4 to 99.3) and interpretation of the results (94.7%, 95% CI 90.0 to 97.3).

CONCLUSION

Similar to prediction model studies developed using conventional regression-based techniques, the completeness of reporting is poor. Essential information to decide to use the model (i.e. model specification and its performance) is rarely reported. However, some items and sub-items of TRIPOD might be less suitable for ML-based prediction model studies and thus, TRIPOD requires extensions. Overall, there is an urgent need to improve the reporting quality and usability of research to avoid research waste.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO, CRD42019161764.

Reporting of abstracts in studies that used routinely collected data for exploring drug treatment effects: a cross-sectional survey

Background

In recent years, studies that used routinely collected data (RCD), such as electronic medical records and administrative claims, for exploring drug treatment effects, including effectiveness and safety, have been increasingly published. Abstracts of such studies represent a highly attended source for busy clinicians or policy-makers, and are important for indexing by literature database. If less clearly presented, they may mislead decisions or indexing. We thus conducted a cross-sectional survey to systematically examine how the abstracts of such studies were reported.

Methods

We searched PubMed to identify all observational studies published in 2018 that used RCD for assessing drug treatment effects. Teams of methods-trained collected data from eligible studies using pilot-tested, standardized forms that were developed and expanded from “The reporting of studies conducted using observational routinely collected health data statement for pharmacoepidemiology” (RECORD-PE) statement. We used descriptive analyses to examine how authors reported data source, study design, data analysis, and interpretation of findings.

Results

A total of 222 studies were included, of which 118 (53.2%) reported type of database used, 17 (7.7%) clearly reported database linkage, and 140 (63.1%) reported coverage of data source. Only 44 (19.8%) studies stated a predefined hypothesis, 127 (57.2%) reported study design, 140 (63.1%) reported statistical models used, 142 (77.6%) reported adjusted estimates, 33 (14.9%) mentioned sensitivity analyses, and 39 (17.6%) made a strong claim about treatment effect. Studies published in top 5 general medicine journals were more likely to report the name of data source (94.7% vs. 67.0%) and study design (100% vs. 53.2%) than those in other journals.

Conclusions

The under-reporting of key methodological features in abstracts of RCD studies was common, which would substantially compromise the indexing of this type of literature and prevent the effective use of study findings. Substantial efforts to improve the reporting of abstracts in these studies are highly warranted.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01482-9.

Comparison between 2000 and 2018 on the reporting of statistical significance and clinical relevance in physiotherapy clinical trials in six major physiotherapy journals: a meta-research design

DESIGN

Meta-research.

OBJECTIVE

To compare the prevalence of reporting p values, effect estimates and clinical relevance in physiotherapy randomised controlled trials (RCTs) published in the years 2000 and 2018.

METHODS

We performed a meta-research study of physiotherapy RCTs obtained from six major physiotherapy peer-reviewed journals that were published in the years 2000 and 2018. We searched the databases Embase, Medline and PubMed in May 2019, and extracted data on the study characteristics and whether articles reported on statistical significance, effect estimates and confidence intervals for baseline, between-group, and within-group differences, and clinical relevance. Data were presented using descriptive statistics and inferences were made based on proportions. A 20% difference between 2000 and 2018 was regarded as a meaningful difference.

RESULTS

We found 140 RCTs: 39 were published in 2000 and 101 in 2018. Overall, there was a high prevalence (>90%) of reporting p values for the main (between-group) analysis, with no difference between years. Statistical significance testing was frequently used for evaluating baseline differences, increasing from 28% in 2000 to 61.4% in 2018. The prevalence of reporting effect estimates, CIs and the mention of clinical relevance increased from 2000 to 2018 by 26.6%, 34% and 32.8% respectively. Despite an increase in use in 2018, over 40% of RCTs failed to report effect estimates, CIs and clinical relevance of results.

CONCLUSION

The prevalence of using p values remains high in physiotherapy research. Although the proportion of reporting effect estimates, CIs and clinical relevance is higher in 2018 compared to 2000, many publications still fail to report and interpret study findings in this way.

Rigour and reproducibility in Canadian research: call for a coordinated approach

Shortcomings in the rigour and reproducibility of research have become well-known issues and persist despite repeated calls for improvement. A coordinated effort among researchers, institutions, funders, publishers, learned societies, and regulators may be the most effective way of tackling these issues. The UK Reproducibility Network (UKRN) has fostered collaboration across various stakeholders in research and are creating the infrastructure necessary to advance rigorous and reproducible research practices across the United Kingdom. Other Reproducibility Networks, modelled on UKRN, are now emerging in other countries. Canada could benefit from a comparable network to unify the voices around research quality and maximize the value of Canadian research.

A Scoping Review of 4 Decades of Outcomes in Nonsurgical Root Canal Treatment, Nonsurgical Retreatment, and Apexification Studies-Part 1: Process and General Results

INTRODUCTION

Despite initiatives to standardize reports, variances in study design, outcomes assessed, and tools used are persistent in the literature. This review scoped the existing literature on endodontic outcome studies for future development of core outcome sets.

METHODS

A comprehensive literature search of randomized controlled trials, cohort studies, case-control studies, and case series (≥10 patients) published after 1980 including patients ≥10 years of age with any preoperative pulpal and periapical diagnosis in permanent teeth requiring nonsurgical root canal treatment (NS-RCT), retreatment (NS-ReTx), or apexification was performed. Abstracted data were reported through descriptive statistics.

RESULTS

Of the 9957 studies screened, 354 were included. An increase in the quantity of endodontic outcome publications and levels of evidence in their study design was noted over the past 4 decades. Although 41% of the studies included participants 26-50 years of age, literature including participants >50 years old has increased since 2000. Apexification and NS-ReTx were mostly provided by specialists and postgraduate students. The most common follow-up period was 2-5 years (35%), and most randomized controlled trials (58%) reported follow-up times <1 year. Multiple-visit treatment was most common in apexification studies (85%). Deficiency, inconsistency, and ambiguity were observed across many reports.

CONCLUSIONS

NS-ReTx and NS-RCT/NS-ReTx studies have increased over the past 2 decades, particularly those focusing on molars and patients >50 years old. Despite the progress in endodontic research, heterogeneity in reporting styles yields considerable limitations, particularly data standardization challenges and inconsistencies in methods and results reporting. This scoping review highlighted the state of available research and supported the development of standardized guidelines for future investigations.

Application of the Templates TIDieR and CERT Reveal Incomplete Reporting and Poor Replicability of Exercise Interventions for Type 2 Diabetes Mellitus

BACKGROUND

Exercise is strongly recommended for the management of type 2 diabetes mellitus (T2DM). However, incomplete intervention reporting in clinical trials limits the replication of exercise protocols. As previously demonstrated by us for exercise and hypertension, the reporting quality might also be insufficient in studies with respect to T2DM and exercise.

OBJECTIVE

The aim of the study was to assess the completeness of exercise intervention reporting in randomized controlled trials (RCTs) for T2DM.

METHODS

Two independent reviewers applied the Consensus on Exercise Reporting Template (CERT) and the template for intervention description and replication (TIDieR) to 23 exercise trials obtained from the most recent and frequently cited meta-analysis in current guidelines. The completeness of reporting was evaluated, focusing on the F.I.T.T. components (frequency, intensity, time, type). Interrater agreement and associations with publication year and journal impact factor were examined.

RESULTS

Mean CERT score was 11/19 (range 5-17), and 8/12 (range 4-12) for TIDieR. F.I.T.T. components were almost completely described, whereas overall completeness of exercise reporting was 60% and 68% (CERT and TIDieR). Replication of each exercise of the respective program was not possible in 52% of interventions. The majority of items had shown excellent agreement. No associations with publication year or impact factor were found.

CONCLUSION

Exercise interventions were not found to be sufficiently reported in RCTs that currently guide clinical practice in T2DM. Replication in further studies or clinical practice is limited due to poor exercise description. We suggest the use of more specific CERT for reporting results of exercise interventions. Further refinement for internal diseases is needed to better describe exercise interventions.

Using a Digital Twin of an Electrical Stimulation Device to Monitor and Control the Electrical Stimulation of Cells in vitro

Electrical stimulation for application in tissue engineering and regenerative medicine has received increasing attention in recent years. A variety of stimulation methods, waveforms and amplitudes have been studied. However, a clear choice of optimal stimulation parameters is still not available and is complicated by ambiguous reporting standards. In order to understand underlying cellular mechanisms affected by the electrical stimulation, the knowledge of the actual prevailing field strength or current density is required. Here, we present a comprehensive digital representation, a digital twin, of a basic electrical stimulation device for the electrical stimulation of cells in vitro. The effect of electrochemical processes at the electrode surface was experimentally characterised and integrated into a numerical model of the electrical stimulation. Uncertainty quantification techniques were used to identify the influence of model uncertainties on relevant observables. Different stimulation protocols were compared and it was assessed if the information contained in the monitored stimulation pulses could be related to the stimulation model. We found that our approach permits to model and simulate the recorded rectangular waveforms such that local electric field strengths become accessible. Moreover, we could predict stimulation voltages and currents reliably. This enabled us to define a controlled stimulation setting and to identify significant temperature changes of the cell culture in the monitored voltage data. Eventually, we give an outlook on how the presented methods can be applied in more complex situations such as the stimulation of hydrogels or tissue in vivo.

Are orthodontic randomised controlled trials justified with a citation of an appropriate systematic review?

Background

A systematic review of the evidence should be undertaken to support the justification for undertaking a clinical trial. The aim of this study was to examine whether reports of orthodontic Randomised Clinical Trials (RCTs) cite prior systematic reviews (SR) to explain the rationale or justification of the trial. Study characteristics that predicated the citation of SR in the RCT report were also explored.

Material and methods

Orthodontic RCTs published between 1st January 2010 to 31st December 2020 in seven orthodontic journals were identified. All titles and abstracts were screened independently by two authors. Descriptive statistics and associations were assessed for the study characteristics. Logistic regression was used to identify predicators of SR inclusion in the trial report.

Results

301 RCTs fulfilling the eligibility criteria were assessed. 220 SRs were available of which 74.5% (N = 164) were cited, and 24.5% (N = 56) were not included but were available in the literature within 12 months of trial commencement. When a SR was not included in the introduction or no SR was available within 12 months of trial commencement, interventional studies were commonly cited. The continent of the corresponding author predicated the possibility of inclusion of a SR in the introduction (OR 0.36; 95% CI 0.18–0.71; p = 0.003).

Conclusions

A quarter of orthodontic RCTs (24.5%) included in this study did not cite a SR in the introduction section to justify the rationale of the trial when a relevant SR was available. To reduce research waste and optimal usage of resources, researchers should identify or conduct a systematic review of the evidence to support the rationale and justification of the trial.

Challenges for assessing replicability in preclinical cancer biology

We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication – it is hard to assess whether reported findings are credible.

Challenges for assessing replicability in preclinical cancer biology

We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication - it is hard to assess whether reported findings are credible.

Teaching: confidence, prediction and tolerance intervals in scientific practice: a tutorial on binary variables

BACKGROUND

One of the emerging themes in epidemiology is the use of interval estimates. Currently, three interval estimates for confidence (CI), prediction (PI), and tolerance (TI) are at a researcher's disposal and are accessible within the open access framework in R. These three types of statistical intervals serve different purposes. Confidence intervals are designed to describe a parameter with some uncertainty due to sampling errors. Prediction intervals aim to predict future observation(s), including some uncertainty present in the actual and future samples. Tolerance intervals are constructed to capture a specified proportion of a population with a defined confidence. It is well known that interval estimates support a greater knowledge gain than point estimates. Thus, a good understanding and the use of CI, PI, and TI underlie good statistical practice. While CIs are taught in introductory statistical classes, PIs and TIs are less familiar.

RESULTS

In this paper, we provide a concise tutorial on two-sided CI, PI and TI for binary variables. This hands-on tutorial is based on our teaching materials. It contains an overview of the meaning and applicability from both a classical and a Bayesian perspective. Based on a worked-out example from veterinary medicine, we provide guidance and code that can be directly applied in R.

CONCLUSIONS

This tutorial can be used by others for teaching, either in a class or for self-instruction of students and senior researchers.

Using the Barthel Index and modified Rankin Scale as Outcome Measures for Stroke Rehabilitation Trials; A Comparison of Minimum Sample Size Requirements

OBJECTIVES

Underpowered trials risk inaccurate results. Recruitment to stroke rehabilitation randomised controlled trials (RCTs) is often a challenge. Statistical simulations offer an important opportunity to explore the adequacy of sample sizes in the context of specific outcome measures. We aimed to examine and compare the adequacy of stroke rehabilitation RCT sample sizes using the Barthel Index (BI) or modified Rankin Scale (mRS) as primary outcomes.

METHODS

We conducted computer simulations using typical experimental event rates (EER) and control event rates (CER) based on individual participant data (IPD) from stroke rehabilitation RCTs. Event rates are the proportion of participants who experienced clinically relevant improvements in the RCT experimental and control groups. We examined minimum sample size requirements and estimated the number of participants required to achieve a number needed to treat within clinically acceptable boundaries for the BI and mRS.

RESULTS

We secured 2350 IPD (18 RCTs). For a 90% chance of statistical accuracy on the BI a rehabilitation RCT would require 273 participants per randomised group. Accurate interpretation of effect sizes would require 1000s of participants per group. Simulations for the mRS were not possible as a clinically relevant improvement was not detected when using this outcome measure.

CONCLUSIONS

Stroke rehabilitation RCTs with large sample sizes are required for accurate interpretation of effect sizes based on the BI. The mRS lacked sensitivity to detect change and thus may be unsuitable as a primary outcome in stroke rehabilitation trials.

Methodological quality and reporting standards in systematic reviews with meta-analysis of physical activity studies: a report from the Strengthening the Evidence in Exercise Sciences Initiative (SEES Initiative)

BACKGROUND

Several resources have been developed (e.g., reporting guidelines) to promote high-standard practices in health research. However, there was no continuous and systematic assessment of recommended practices in published systematic reviews with meta-analysis (SRMAs), which increases the usability of the available resources. Therefore, we aimed to assess the methodological and reporting standards in SRMAs of physical activity studies. This report presents the main results of the SEES Initiative in 2019.

METHODS

Our approach is based on a prospective systematic review methodology to implement post-publication surveillance of research practices in exercise sciences. Briefly, during the year 2019, pre-specified searches were conducted monthly (PubMed/MEDLINE) in journals from the exercise sciences (n = 9) and medicine (n = 5). The assessments were independently conducted by two authors, based on 36 items/practices derived from established statements/tools (PRISMA, AMSTAR 2, ROBIS). To be eligible, SRMAs should summarize studies that had, at least, one arm consisting of physical activity interventions/exposures and one health or behavioral outcome.

RESULTS

Out of 1028 studies assessed for eligibility, 103 SRMAs were included. The minimum adherence was 13/36 items, whereas only one SRMA adhered to all items. Some highly contemplated items included identification of title as SRMA (97.1%) and descriptions of the main outcome in the abstract (95.1%) and risk of bias (RoB) assessment (95.1%). Some poorly contemplated items included publicly available protocol (4.9%), discussion of the results in light of RoB in studies included (32.0%), and data sharing statements (35.9%).

CONCLUSION

In summary, there is a suboptimal adherence to recommended practices on methodological quality and reporting standards in the SRMAs of physical activity intervention/exposure evaluated from the selected journals in 2019, which likely reduce the reproducibility and usefulness of these studies. This incipient evidence from our first 12 months of post-publication surveillance should serve as a call for attention and action for multiple stakeholders (e.g., authors, reviewers, editors, funders, academic institutions) in this important health research field.

Practical Tools and Guidelines for Young Oncologists From Resource-Limited Settings to Publish Excellence and Advance Their Career

Cancer research is evolving worldwide. However, publishing high-quality academic literature in oncology remains challenging for authors in the developing world. Young oncologists in low- and middle-income countries experience several barriers including lack of funding and research facilities, as well as inadequate training. Publication best practices, science integrity, and ethics are required to improve oncology research quality and therefore, improve patients' care in these countries. To achieve this goal, we propose some basic principles and tools that may help young oncologists especially in developing countries overcome these issues and boost their academic careers.

Epidemiology, methodological quality, and reporting characteristics of systematic reviews and meta-analyses on coronavirus disease 2019: A cross-sectional study

During the coronavirus disease 2019 (COVID-19) pandemic, convenient accessibility and rapid publication of studies related to the ongoing pandemic prompted shorter preparation time for studies. Whether the methodological quality and reporting characteristics of published systematic reviews (SRs)/meta-analyses are affected during the specific pandemic condition is yet to be clarified. This study aimed to evaluate the epidemiology, methodological quality, and reporting characteristics of published SRs/meta-analyses related to COVID-19.The Ovid Medline, Ovid Embase, Cochrane Library, and Web of Science electronic databases were searched to identify published SRs/meta-analyses related to the COVID-19 pandemic. Study screening, data extraction, and methodology quality assessment were performed independently by 2 authors. The methodology quality of included SRs/meta-analyses was evaluated using revised version of a measurement tool to assess SRs, and the reporting characteristics were assessed based on the preferred reporting items for SRs and meta-analyses guidelines.A total of 47 SRs/meta-analyses were included with a low to critically low methodological quality. The median number of days from the date of literature retrieval to the date that the study was first available online was 21 days; due to the limited time, only 7 studies had study protocols, and the studies focused on a wide range of COVID-19 topics. The rate of compliance to the preferred reporting items for SRs and meta-analyses checklists of reporting characteristics ranged from 14.9% to 100%. The rate of compliance to the items of protocol and registration, detailed search strategy, and assessment of publication bias was less than 50%.SRs/meta-analyses on COVID-19 were poorly conducted and reported, and thus, need to be substantially improved.

Technology-Based Interventions for Cancer Caregivers: Concept Analysis

Background

Cancer is a taxing chronic disease that demands substantial care, most of which is shouldered by informal caregivers. As a result, cancer caregivers often have to manage considerable challenges that could result in severe physical and psychological health consequences. Technology-based interventions have the potential to address many, if not all, of the obstacles caregivers encounter while caring for patients with cancer. However, although the application of technology-based interventions is on the rise, the term is seldom defined in research or practice. Considering that the lack of conceptual clarity of the term could compromise the effectiveness of technology-based interventions for cancer caregivers, timely research is needed to bridge this gap.

Objective

This study aims to clarify the meaning of technology-based interventions in the context of cancer caregiving and provide a definition that can be used by cancer caregivers, patients, clinicians, and researchers to facilitate evidence-based research and practice.

Methods

The 8-step concept analysis method by Walker and Avant was used to analyze the concept of technology-based interventions in the context of cancer caregiving. PubMed, PsycINFO, CINAHL, and Scopus were searched for studies that examined technology-based interventions for cancer caregivers.

Results

The defining attributes of technology-based interventions were recognized as being accessible, affordable, convenient, and user-friendly. On the basis of insights gained on the defining attributes, antecedents to, and consequences of technology-based interventions through the concept analysis process, technology-based interventions were defined as the use of technology to design, develop, and deliver health promotion contents and strategies aimed at inducing or improving positive physical or psychological health outcomes in cancer caregivers.

Conclusions

This study clarified the meaning of technology-based interventions in the context of cancer caregiving and provided a clear definition that can be used by caregivers, patients, clinicians, and researchers to facilitate evidence-based oncology practice. A clear conceptualization of technology-based interventions lays foundations for better intervention design and research outcomes, which in turn have the potential to help health care professionals address the needs and preferences of cancer caregivers more cost-effectively.

Developing Core Outcome Sets (COS) and Core Outcome Measures Sets (COMS) in Cosmetic Gynecological Interventions: Protocol for a Development and Usability Study

Background

Studies evaluating cosmetic gynecological interventions have followed variable methodology and reported a diversity of outcomes. Such variations limit the comparability of studies and the value of research-based evidence. The development of core outcome sets (COS) and core outcome measures sets (COMS) would help address these issues, ensuring a minimum of outcomes important to all stakeholders, primarily women requesting or having experienced cosmetic gynecological interventions.

Objective

This protocol describes the methods used in developing a COS and COMS for cosmetic gynecological interventions.

Methods

An international steering group within CHORUS, including health care professionals, researchers, and women with experience in cosmetic gynecological interventions from 4 continents, will guide the development of COS and COMS. Potential outcome measures and outcomes will be identified through comprehensive literature reviews. These potential COS and COMS will be entered into an international, multi-perspective web-based Delphi survey where Delphi participants judge which domains will be core. A priori thresholds for consensus will get established before each Delphi round. The Delphi survey results will be evaluated quantitatively and qualitatively in subsequent stakeholder group consensus meetings in the process of establishing “core” outcomes.

Results

Dissemination and implementation of the resulting COS and COMS within an international context will be promoted and reviewed.

Conclusions

This protocol presents the steps in developing a COS and COMS for cosmetic gynecological interventions. Embedding the COS and COMS for cosmetic gynecological interventions within future clinical trials, systematic reviews, and practice guidelines could contribute to enhancing the value of research and improving overall patient care.

Trial Registration

Core Outcome Measures in Effectiveness Trials (COMET) 1592; https://tinyurl.com/n8faysuh

International Registered Report Identifier (IRRID)

PRR1-10.2196/28032

Use of core outcome sets was low in clinical trials published in major medical journals

OBJECTIVES

To examine current practices in late-phase trials published in major medical journals and examine trialists' views about core outcome set (COS) use.

STUDY DESIGN AND SETTING

A sequential multi-methods study was conducted. We examined late-phase trials published between October 2019 and March 2020 in JAMA, NEJM, The Lancet, BMJ, and Annals of Internal Medicine. The COMET database was searched for COS potentially relevant to trials not reporting using a COS; overlap of trial and COS outcomes was examined. An online survey examined awareness of, and decisions to search for and use a COS.

RESULTS

Ninety-five trials were examined; 93 (98%) did not report using a COS. Relevant COS were identified for 31 trials (33%). Core outcomes were measured in 9 (23%) studies; all trials measured at least one core outcome. Thirty-one trialists (33%) completed our survey. The most common barrier to COS use was trialist's own outcome preferences and choice (68%). The most common perceived facilitator was awareness and knowledge about COS (90%).

CONCLUSION

COS use in this cohort of trials was low, even when relevant COS were available. Increased use of COS in clinical trials can improve evaluation of intervention effects and evidence synthesis and reduce research waste.

Smaller spared subcortical nuclei are associated with worse post-stroke sensorimotor outcomes in 28 cohorts worldwide

Up to two-thirds of stroke survivors experience persistent sensorimotor impairments. Recovery relies on the integrity of spared brain areas to compensate for damaged tissue. Deep grey matter structures play a critical role in the control and regulation of sensorimotor circuits. The goal of this work is to identify associations between volumes of spared subcortical nuclei and sensorimotor behaviour at different timepoints after stroke. We pooled high-resolution T1-weighted MRI brain scans and behavioural data in 828 individuals with unilateral stroke from 28 cohorts worldwide. Cross-sectional analyses using linear mixed-effects models related post-stroke sensorimotor behaviour to non-lesioned subcortical volumes (Bonferroni-corrected, P < 0.004). We tested subacute (≤90 days) and chronic (≥180 days) stroke subgroups separately, with exploratory analyses in early stroke (≤21 days) and across all time. Sub-analyses in chronic stroke were also performed based on class of sensorimotor deficits (impairment, activity limitations) and side of lesioned hemisphere. Worse sensorimotor behaviour was associated with a smaller ipsilesional thalamic volume in both early (n = 179; d = 0.68) and subacute (n = 274, d = 0.46) stroke. In chronic stroke (n = 404), worse sensorimotor behaviour was associated with smaller ipsilesional putamen (d = 0.52) and nucleus accumbens (d = 0.39) volumes, and a larger ipsilesional lateral ventricle (d = -0.42). Worse chronic sensorimotor impairment specifically (measured by the Fugl-Meyer Assessment; n = 256) was associated with smaller ipsilesional putamen (d = 0.72) and larger lateral ventricle (d = -0.41) volumes, while several measures of activity limitations (n = 116) showed no significant relationships. In the full cohort across all time (n = 828), sensorimotor behaviour was associated with the volumes of the ipsilesional nucleus accumbens (d = 0.23), putamen (d = 0.33), thalamus (d = 0.33) and lateral ventricle (d = -0.23). We demonstrate significant relationships between post-stroke sensorimotor behaviour and reduced volumes of deep grey matter structures that were spared by stroke, which differ by time and class of sensorimotor measure. These findings provide additional insight into how different cortico-thalamo-striatal circuits support post-stroke sensorimotor outcomes.

Reporting Policies in Neurosurgical Journals: A Meta-Science Study of the Current State and Case for Standardization

BACKGROUND

Reporting quality within the neurosurgical literature is low, limiting the ability of journals to act as gatekeepers for evidence-based neurosurgical care. Journal policies during article submission aim to improve reporting quality. We conducted a meta-science study characterizing the reporting policies of neurosurgical journals and other related peer-reviewed publications.

METHODS

Journals were retrieved in 7 searches using Journal Citation Reports and Google Scholar. Characteristics, impact metrics, and submission policies were extracted.

RESULTS

Of 486 results, 54 journals were included, including 27 neurosurgical and 27 related topical journals. Thirty-eight (70.4%) adopted authorship guidelines and 20 (37.0%) disclosure standards of the International Council of Medical Journal Editors. Twenty-six (48.1%) required data availability statement and 33 (61.1%) clinical trials registration. Twenty-one (38.9%) required and 11 (20.4%) recommended adherence to reporting guidelines. Twenty (37.0%) endorsed EQUATOR network guidelines. PRISMA was mentioned by 30 (55.6%) journals, CONSORT by 28 (51.9%), and STROBE by 18 (33.3%). Among neurosurgical journals, factors associated with a requirement or recommendation to follow reporting guidelines among neurosurgical journals included impact factor (P = 0.0013), Article Influence Score (P = 0.0236), SCImago h-index (P = 0.0152), SCImago journal rank (P = 0.002), and CiteScore (P = 0.0023), as well as recommendations pertaining to International Council of Medical Journal Editors authorship guidelines (P = 0.0085), ORCID (P = 0.014), clinical trials registration (P = 0.0369), or data availability statement (P = 0.0047). CONSORT, PRISMA, or STROBE delineations were significantly associated with the mention of another guideline (P < 0.01).

CONCLUSIONS

Neurosurgical journal submission policies are inconsistent. Frameworks to improve reporting quality are uncommonly used. Increasing rigor and standardization of reporting policies across journals publishers may improve quality.

Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review

OBJECTIVE

To assess the methodological quality of studies on prediction models developed using machine learning techniques across all medical specialties.

DESIGN

Systematic review.

DATA SOURCES

PubMed from 1 January 2018 to 31 December 2019.

ELIGIBILITY CRITERIA

Articles reporting on the development, with or without external validation, of a multivariable prediction model (diagnostic or prognostic) developed using supervised machine learning for individualised predictions. No restrictions applied for study design, data source, or predicted patient related health outcomes.

REVIEW METHODS

Methodological quality of the studies was determined and risk of bias evaluated using the prediction risk of bias assessment tool (PROBAST). This tool contains 21 signalling questions tailored to identify potential biases in four domains. Risk of bias was measured for each domain (participants, predictors, outcome, and analysis) and each study (overall).

RESULTS

152 studies were included: 58 (38%) included a diagnostic prediction model and 94 (62%) a prognostic prediction model. PROBAST was applied to 152 developed models and 19 external validations. Of these 171 analyses, 148 (87%, 95% confidence interval 81% to 91%) were rated at high risk of bias. The analysis domain was most frequently rated at high risk of bias. Of the 152 models, 85 (56%, 48% to 64%) were developed with an inadequate number of events per candidate predictor, 62 handled missing data inadequately (41%, 33% to 49%), and 59 assessed overfitting improperly (39%, 31% to 47%). Most models used appropriate data sources to develop (73%, 66% to 79%) and externally validate the machine learning based prediction models (74%, 51% to 88%). Information about blinding of outcome and blinding of predictors was, however, absent in 60 (40%, 32% to 47%) and 79 (52%, 44% to 60%) of the developed models, respectively.

CONCLUSION

Most studies on machine learning based prediction models show poor methodological quality and are at high risk of bias. Factors contributing to risk of bias include small study size, poor handling of missing data, and failure to deal with overfitting. Efforts to improve the design, conduct, reporting, and validation of such studies are necessary to boost the application of machine learning based prediction models in clinical practice.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42019161764.

Conducting Digital Health Care Research: Document Analysis of Challenges Experienced During Intervention Development and Feasibility Study Setup of an Internet-Administered Intervention for Parents of Children Treated for Cancer

Background

The design and conduct of research to develop, test, and evaluate complex health care interventions is challenging. Although the existing literature describes key challenges associated with the design and conduct of definitive (evaluation) trials, there is a lack of information concerning specific challenges associated with the intervention development phase and setup of feasibility studies. In particular, the literature is scarce concerning the challenges associated with conducting digital health care research, such as research on internet-administered interventions and research using digital features to support the execution of study procedures (eg, recruitment, consent, retention, and data collection and management). This study is conducted in the context of the intervention development and feasibility study setup phases of an internet-administered, guided, low-intensity cognitive behavioral therapy–based intervention for parents of children previously treated for cancer.

Objective

The aim of this study is to explore the challenges experienced during the development phase of the internet-administered intervention and digital features to support the execution of the study procedures and a feasibility study setup.

Methods

To explore the key challenges experienced, we conducted a document analysis of written records from all study meetings held by the research team (meeting minutes) between June 7, 2018, and January 10, 2020, guided by a thematic analysis approach. Furthermore, discussion groups with members of the research team were held to develop a more detailed understanding of the key challenges experienced. Methods and results are reported in accordance with the relevant items from the Standards for Reporting Qualitative Research checklist.

Results

Six main themes were identified: decision-making and communication, expertise, external constraints, flexibility, planning and scheduling, and technical constraints.

Conclusions

Significant challenges were experienced during the intervention development and setup phases of the feasibility study. Implications are discussed to inform future design, conduct, and planning of internet-administered intervention development and feasibility studies, especially within the context of digital health care research.

Systematic review and meta-analyses of studies analysing instructions to authors from 1987 to 2017

To gain insight into changes of scholarly journals' recommendations, we conducted a systematic review of studies that analysed journals' Instructions to Authors (ItAs). We summarised results of 153 studies, and meta-analysed how often ItAs addressed: 1) authorship, 2) conflicts of interest, 3) data sharing, 4) ethics approval, 5) funding disclosure, and 6) International Committee of Medical Journal Editors' Uniform Requirements for Manuscripts. For each topic we found large between-study heterogeneity. Here, we show six factors that explained most of that heterogeneity: 1) time (addressing of topics generally increased over time), 2) country (large differences found between countries), 3) database indexation (large differences found between databases), 4) impact factor (topics were more often addressed in highest than in lowest impact factor journals), 5) discipline (topics were more often addressed in Health Sciences than in other disciplines), and 6) sub-discipline (topics were more often addressed in general than in sub-disciplinary journals).

Reporting of prognostic clinical prediction models based on machine learning methods in oncology needs to be improved

OBJECTIVE

Evaluate the completeness of reporting of prognostic prediction models developed using machine learning methods in the field of oncology.

STUDY DESIGN AND SETTING

We conducted a systematic review, searching the MEDLINE and Embase databases between 01/01/2019 and 05/09/2019, for non-imaging studies developing a prognostic clinical prediction model using machine learning methods (as defined by primary study authors) in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement to assess the reporting quality of included publications. We described overall reporting adherence of included publications and by each section of TRIPOD.

RESULTS

Sixty-two publications met the inclusion criteria. 48 were development studies and 14 were development with validation studies. 152 models were developed across all publications. Median adherence to TRIPOD reporting items was 41% [range: 10%-67%] and at least 50% adherence was found in 19% (n=12/62) of publications. Adherence was lower in development only studies (median: 38% [range: 10%-67%]); and higher in development with validation studies (median: 49% [range: 33%-59%]).

CONCLUSION

Reporting of clinical prediction models using machine learning in oncology is poor and needs urgent improvement, so readers and stakeholders can appraise the study methods, understand study findings, and reduce research waste.

Poor reporting of physical activity and exercise interventions in youth mental health trials: A brief report

AIM

To describe the quality and completeness of the description and reporting of physical activity and exercise interventions delivered to young people to promote mental health or treat mental illness.

METHODS

We conducted a series of scoping reviews identifying 64 controlled trials of physical activity and exercise interventions delivered to young people. We extracted: intervention characteristics, personnel and delivery format, the intensity, duration, frequency and type of physical activity or exercise.

RESULTS

There was limited reporting of intervention details across studies; 52% did not provide information to confidently assess intervention intensity, 29% did not state who delivered the intervention, and 44% did not specify the intervention delivery format.

CONCLUSIONS

We recommend that authors adhere to the CONSORT reporting requirements and its intervention reporting extensions, (a) the Template for Intervention Description and Replication, (b) Consensus for Exercise Reporting Template and (c) as part of this, detail the frequency, intensity, time and type of physical activity recommendations and prescriptions. Without this, future trials are unable to replicate and extend previous work to support or disconfirm existing knowledge, leading to research waste and diminishing translation and implementation potential.

Methodology over metrics: current scientific standards are a disservice to patients and society

Covid-19 research made it painfully clear that the scandal of poor medical research, as denounced by Altman in 1994, persists today. The overall quality of medical research remains poor, despite longstanding criticisms. The problems are well known, but the research community fails to properly address them. We suggest that most problems stem from an underlying paradox: although methodology is undeniably the backbone of high-quality and responsible research, science consistently undervalues methodology. The focus remains more on the destination (research claims and metrics) than on the journey. Notwithstanding, research should serve society more than the reputation of those involved. While we notice that many initiatives are being established to improve components of the research cycle, these initiatives are too disjointed. The overall system is monolithic and slow to adapt. We assert that top-down action is needed from journals, universities, funders and governments to break the cycle and put methodology first. These actions should involve the widespread adoption of registered reports, balanced research funding between innovative, incremental and methodological research projects, full recognition and demystification of peer review, improved methodological review of reports, adherence to reporting guidelines, and investment in methodological education and research. Currently, the scientific enterprise is doing a major disservice to patients and society.

High prevalence of spin was found in pharmacovigilance studies using disproportionality analyses to detect safety signals: a meta-epidemiological study

OBJECTIVE

To systematically review and appraise misinterpretation of pharmacovigilance disproportionality analysis results in published studies.

STUDY DESIGN AND SETTING

We randomly selected 100 studies that performed disproportionality analyses and indexed in Medline identified during a systematic literature search. Titles, abstracts and main texts (results, discussion and conclusion) were evaluated for spin independently by two reviewers. Spin in pharmacovigilance studies was classified according to three main categories: inappropriate interpretation, inappropriate extrapolations and misleading reporting.

RESULTS

Of the 100 studies evaluated, we found that 63%, 56% and 51% had at least one type of spin in their abstract, main text or conclusion respectively, and 40% used causal language to interpret their results in the abstract or conclusion. Spin in titles and results were exclusively represented by inappropriate interpretations of findings (12% and 21% respectively), with terms such as "risk of" or "risks associated with" or results erroneously presented as regular Odds Ratios. Spin in discussion sections mostly concerned inappropriate interpretations (38%)and misleading reporting (12%). Misleading reporting, notably failing to acknowledge the limitations of disproportionality analyses, was the most frequent type of spin in abstracts (55%) and conclusion sections (37%).

CONCLUSION

We found that spin is frequent in publications of pharmacovigilance disproportionality analyses, notably in abstracts. This consisted notably in an over-interpretation of the results suggesting a proven causative link between a drug use and the risk of an event.

The effectiveness of interventions to increase patient involvement in decision-making in orthodontics: A systematic review

OBJECTIVES

To examine the effectiveness of interventions that aim to increase patient involvement in treatment decisions in orthodontic, orthognathic and cleft treatment, based on patient-reported outcomes and patient knowledge.

DESIGN

Systematic review.

DATA SOURCES

OVID databases (MEDLINE, EMBASE and EBM reviews), CENTRAL, WHO's International Clinical Trials Registry Platform and reference lists of included studies.

DATA SELECTION

Studies were selected by two reviewers independently and in duplicate based on pre-defined eligibility criteria: Population: People considering or undergoing orthodontic, orthognathic or cleft treatment. Intervention: Any intervention that aims to increase patient involvement in decision-making. Outcomes: Patient-reported outcomes and patient knowledge. Studies: All experimental studies published in English from January 2000 to October 2019 were eligible.

DATA EXTRACTION

Standardised data extraction of study information and assessment of risk of bias using the Cochrane Risk of Bias Tool for RCTs and ROBINS-I for non-randomised studies of interventions.

DATA SYNTHESIS

13 randomised controlled trials were included. Due to heterogeneity in the studies, a narrative synthesis was undertaken. The majority (n=11) of studies involved orthodontic patients, with one study of cleft patients and one study of orthognathic and orthodontic patients. Six included studies reported significant differences between intervention and control groups with improved patient knowledge or better patient-reported outcomes.

CONCLUSIONS

A variety of different interventions and outcome measures were used making data synthesis challenging. There is some evidence that interventions to increase patient involvement in decision-making can improve patient-reported outcomes and patient knowledge.

Discrepancies between Conference Abstracts and Published Manuscripts in Plastic Surgery Studies: A Retrospective Review

Background

Inconsistency in results and outcomes between presented abstracts and corresponding published articles can negatively affect clinical education and care. The objective of this study was to describe the frequency of clinically meaningful change in results and outcomes between abstracts presented at the American Association of Plastic Surgeons annual conference and the corresponding published articles, and to determine risk factors associated with discrepancies.

Methods

All abstracts delivered as oral presentations at the American Association of Plastic Surgeons conference (2006-2016) were reviewed. Results and outcomes were compared with those in corresponding articles. We defined clinically meaningful discrepancy as any change in the directionality of an outcome, or a quantitative change in results exceeding 10%.

Results

Four hundred eighty-six abstracts were identified. Of these, 63% (N = 305) advanced to publication. Of the published studies, 19% (N = 59) contained a discrepancy. In 85% of these (N = 50), discrepancies could not be explained by random variation. Changes in sample size were associated with heightened risk for a discrepancy (OR 10.38, 95% CI 5.16-20.86, P < 0.001). A decrease in sample size greater than 10% increased the likelihood of a discrepancy by 25-fold (OR 24.92, 95% CI 8.66-71.68, P < 0.001), whereas an increase in sample size greater than 10% increased the likelihood of a discrepancy by eight-fold (OR 8.36, CI 3.69-19.00, P < 0.001).

Conclusions

Most discrepancies between abstract and published article were not due to random statistical variation. To mitigate the possible impact of unreliable abstracts, we recommend abstracts be marked as preliminary, that authors indicate whether sample size is final at time of presentation, and that changes to previously reported results be indicated in final publications.

Improving Social Justice in COVID-19 Health Research: Interim Guidelines for Reporting Health Equity in Observational Studies

The COVID-19 pandemic has highlighted the global imperative to address health inequities. Observational studies are a valuable source of evidence for real-world effects and impacts of implementing COVID-19 policies on the redistribution of inequities. We assembled a diverse global multi-disciplinary team to develop interim guidance for improving transparency in reporting health equity in COVID-19 observational studies. We identified 14 areas in the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) checklist that need additional detail to encourage transparent reporting of health equity. We searched for examples of COVID-19 observational studies that analysed and reported health equity analysis across one or more social determinants of health. We engaged with Indigenous stakeholders and others groups experiencing health inequities to co-produce this guidance and to bring an intersectional lens. Taking health equity and social determinants of health into account contributes to the clinical and epidemiological understanding of the disease, identifying specific needs and supporting decision-making processes. Stakeholders are encouraged to consider using this guidance on observational research to help provide evidence to close the inequitable gaps in health outcomes.

Challenges and Lessons Learned From COVID-19 Trials: Should We Be Doing Clinical Trials Differently?

The COVID-19 crisis led to a flurry of clinical trials activity. The COVID-evidence database shows 2814 COVID-19 randomized trials registered as of February 16, 2021. Most were small (only 18% have a planned sample size > 500) and the rare completed ones have not provided published results promptly (only 283 trial publications as of February 2021). Small randomized trials and observational, nonrandomized analyses have not had a successful track record and have generated misleading expectations. Different large trials on the same intervention have generally been far more efficient in producing timely and consistent evidence. The rapid generation of evidence and accelerated dissemination of results have led to new challenges for systematic reviews and meta-analyses (eg, rapid, living, and scoping reviews). Pressure to regulatory agencies has also mounted with massive emergency authorizations, but some of them have had to be revoked. Pandemic circumstances have disrupted the way trials are conducted; therefore, new methods have been developed and adopted more widely to facilitate recruitment, consent, and overall trial conduct. On the basis of the COVID-19 experience and its challenges, planning of several large, efficient trials, and wider use of adaptive designs might change the future of clinical research. Pragmatism, integration in clinical care, efficient administration, promotion of collaborative structures, and enhanced integration of existing data and facilities might be several of the legacies of COVID-19 on future randomized trials.

Quality assessment of case reports in high-impact urology journals using SCARE guideline

BACKGROUND AND AIMS

The usefulness of case reports is dependent on the complete, consistent, and rigorous reporting of these cases. In order to provide a standard guideline for reporting surgical case reports, the SCARE (Surgical CAse REport) guidelines were developed in 2016. The present study evaluated the completeness and transparency of published case reports in high-impact urology journals based on the SCARE guideline.

METHODS

This cross-sectional study was performed on 100 case reports published in Urology, Urology Journal, BMC Urology, and Urology Case Reports journal. Two independent reviewers performed the scoring using the last version of SCARE statement. Each of the 34 items of SCARE guideline were classified as "yes" if the item was reported in the case report text. The SCARE items were classified as "no" when the authors of case reports had not reported that item or could not tell something about reporting the item. Completeness of reporting (COR) score was calculated for each case report. COR score (%) is defined as ["yes" answers/("yes" answers + "no" answers)] × 100 for each case report.

RESULTS

The mean COR score for all the assessed case reports was 49%, ranging from 21% to 79%. Topics with the highest mean COR score were introduction (77% ± 42%), additional information (75% ± 43%), patient information (65% ± 19%), and abstract (66% ± 24%). In contrast, topics with the lowest mean COR were patient perspective (1% ± 10%) and keywords (3% ± 17%).

CONCLUSION

The present study showed that case reports published in urology journals suffer from insufficient reporting. SCARE or CARE guidelines can provide a framework for assessing the reporting quality of case reports before publication. Nevertheless, further studies are highly recommended to better evaluate the efficacy of these guidelines' endorsement on the quality of case reports published in urology journals.

Social Innovation For Health Research: Development of the SIFHR Checklist

BACKGROUND

Social innovations in health are inclusive solutions to address the healthcare delivery gap that meet the needs of end users through a multi-stakeholder, community-engaged process. While social innovations for health have shown promise in closing the healthcare delivery gap, more research is needed to evaluate, scale up, and sustain social innovation. Research checklists can standardize and improve reporting of research findings, promote transparency, and increase replicability of study results and findings.

METHODS AND FINDINGS

The research checklist was developed through a 3-step community-engaged process, including a global open call for ideas, a scoping review, and a 3-round modified Delphi process. The call for entries solicited checklists and related items and was open between November 27, 2019 and February 1, 2020. In addition to the open call submissions and scoping review findings, a 17-item Social Innovation For Health Research (SIFHR) Checklist was developed based on the Template for Intervention Description and Replication (TIDieR) Checklist. The checklist was then refined during 3 rounds of Delphi surveys conducted between May and June 2020. The resulting checklist will facilitate more complete and transparent reporting, increase end-user engagement, and help assess social innovation projects. A limitation of the open call was requiring internet access, which likely discouraged participation of some subgroups.

CONCLUSIONS

The SIFHR Checklist will strengthen the reporting of social innovation for health research studies. More research is needed on social innovation for health.

The evolution of instrument selection for inclusion in core outcome sets at OMERACT: Filter 2.2

INTRODUCTION

OMERACT uses an evidence-based framework known as the 'OMERACT Filter Instrument Selection Algorithm' (OFISA) to guide decisions in the assessment of outcome measurement instruments for inclusion in a core outcome set for interventional and observational clinical trials.

METHODS

A group of OMERACT imaging and patient-centered outcome methodologists worked with imaging outcome groups to facilitate the selection of imaging outcome measurement instruments using the OFISA approach. The lessons learned from this work influenced the evolution to Filter 2.2 and necessitated changes to OMERACT's documentation and processes.

RESULTS

OMERACT has revised documentation and processes to incorporate the evolution of instrument selection to Filter 2.2. These revisions include creation of a template for detailed definitions of the target domain which is a necessary first step for instrument selection, modifications to the Summary of Measurement Properties (SOMP) table to account for sources of variability, and development of standardized reporting tables for each measurement property.

CONCLUSIONS

OMERACT Filter 2.2 represents additional modifications of the OMERACT guide for working groups in their rigorous assessment of measurement properties of instruments of various types, including imaging outcome measurement instruments. Enhanced reporting aims to increase the transparency of the evidence base leading to judgements for the endorsement of instruments in core outcome sets.

Machine learning with neuroimaging data to identify autism spectrum disorder: a systematic review and meta-analysis

PURPOSE

Autism Spectrum Disorder (ASD) is diagnosed through observation or interview assessments, which is time-consuming, subjective, and with questionable validity and reliability. Thus, we aimed to evaluate the role of machine learning (ML) with neuroimaging data to provide a reliable classification of ASD.

METHODS

A systematic search of PubMed, Scopus, and Embase was conducted to identify relevant publications. Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) was used to assess the studies' quality. A bivariate random-effects model meta-analysis was employed to evaluate the pooled sensitivity, the pooled specificity, and the diagnostic performance through the hierarchical summary receiver operating characteristic (HSROC) curve of ML with neuroimaging data in classifying ASD. Meta-regression was also performed.

RESULTS

Forty-four studies (5697 ASD and 6013 typically developing individuals [TD] in total) were included in the quantitative analysis. The pooled sensitivity for differentiating ASD from TD individuals was 86.25 95% confidence interval [CI] (81.24, 90.08), while the pooled specificity was 83.31 95% CI (78.12, 87.48) with a combined area under the HSROC (AUC) of 0.889. Higgins I² (> 90%) and Cochran's Q (p < 0.0001) suggest a high degree of heterogeneity. In the bivariate model meta-regression, a higher pooled specificity was observed in studies not using a brain atlas (90.91 95% CI [80.67, 96.00], p = 0.032). In addition, a greater pooled sensitivity was seen in studies recruiting both males and females (89.04 95% CI [83.84, 92.72], p = 0.021), and combining imaging modalities (94.12 95% [85.43, 97.76], p = 0.036).

CONCLUSION

ML with neuroimaging data is an exciting prospect in detecting individuals with ASD but further studies are required to improve its reliability for usage in clinical practice.

Overall Reporting Descriptions of Acupuncture for Chronic Pain in Randomized Controlled Trials in English Journals

BACKGROUND

Whether the clinical effect of acupuncture in chronic pain is effective has always been a hot topic of research, which has a great relationship with the overall reporting descriptions of acupuncture, especially the sham acupuncture intervention. To confirm the effectiveness of acupuncture, more clinical studies are often required. Therefore, it is necessary to report high-quality and complete descriptions of acupuncture in clinical trials. This study aims to assess the overall reporting quality of acupuncture for chronic pain in randomized controlled trials (RCTs).

METHODS

Three databases from inception to March 2020 were searched, to assess the quality of acupuncture reports included the RCTs based on the pain-specific supplement to Consolidated Standards for Reporting Trials (CONSORT) and Standards for Reporting Interventions in Controlled Trials of Acupuncture (STRICTA) guidelines. The quality of sham acupuncture descriptions was evaluated based on the Template for Intervention Description and Replication (TIDieR)-placebo checklist. Descriptive statistics and analysis of the results were carried out according to the percentage of each item.

RESULTS

A total of 74 RCTs were included which met the inclusion criteria. Based on the pain-specific CONSORT, the reporting rates of "Statistical methods", "Participant flow", and "Blinding" were "52.70%", "70.27%", and "77.03%", respectively. The weakest reported items in STRICTA were related to the depth of insertion (Item 2c, 54.05%) and the setting and context of treatment (Item 4b, 0.00%). Based on the TIDieR-placebo checklist, the reporting rates of "Item 12", "Item 11", "Item 13", "Item 3", and "Item 4" were "8.11%", "10.81%", "29.73%", " 44.59% ", and "47.30%", respectively.

CONCLUSION

At present, the overall report quality of acupuncture treatment for chronic pain in English journals is acceptable, but the report rate in some aspects is still low. In the future, researchers should report RCTs of acupuncture following cleaner checklists and guidelines.

Strengthening the incentives for responsible research practices in Australian health and medical research funding

BACKGROUND

Australian health and medical research funders support substantial research efforts, and incentives within grant funding schemes influence researcher behaviour. We aimed to determine to what extent Australian health and medical funders incentivise responsible research practices.

METHODS

We conducted an audit of instructions from research grant and fellowship schemes. Eight national research grants and fellowships were purposively sampled to select schemes that awarded the largest amount of funds. The funding scheme instructions were assessed against 9 criteria to determine to what extent they incentivised these responsible research and reporting practices: (1) publicly register study protocols before starting data collection, (2) register analysis protocols before starting data analysis, (3) make study data openly available, (4) make analysis code openly available, (5) make research materials openly available, (6) discourage use of publication metrics, (7) conduct quality research (e.g. adhere to reporting guidelines), (8) collaborate with a statistician, and (9) adhere to other responsible research practices. Each criterion was answered using one of the following responses: "Instructed", "Encouraged", or "No mention".

RESULTS

Across the 8 schemes from 5 funders, applicants were instructed or encouraged to address a median of 4 (range 0 to 5) of the 9 criteria. Three criteria received no mention in any scheme (register analysis protocols, make analysis code open, collaborate with a statistician). Importantly, most incentives did not seem strong as applicants were only instructed to register study protocols, discourage use of publication metrics and conduct quality research. Other criteria were encouraged but were not required.

CONCLUSIONS

Funders could strengthen the incentives for responsible research practices by requiring grant and fellowship applicants to implement these practices in their proposals. Administering institutions could be required to implement these practices to be eligible for funding. Strongly rewarding researchers for implementing robust research practices could lead to sustained improvements in the quality of health and medical research.

Does drug dispensing improve the health outcomes of patients attending community pharmacies? A systematic review

BACKGROUND

Drug dispensing is a clinical pharmacy service that promotes access to medicines and their rational use. However, there is a lack of evidence for the impact of drug dispensing on patients' health outcomes. Thus, the purpose of this study was to assess the influence of drug dispensing on the clinical, humanistic, and economic outcomes of patients attending community pharmacies.

METHODS

A systematic literature search was performed in April 2021 using PubMed, Web of Science, Cochrane Library, LILACS, and Open Thesis. Two reviewers screened titles, abstracts, and full-text articles according to the eligibility criteria. Methodological quality was assessed using tools from the Joanna Briggs Institute, and the literature was synthesized narratively.

RESULTS

We retrieved 3,685 articles and included nine studies that presented 13 different outcomes. Regarding the design, they were cross-sectional (n = 4), randomized clinical trials (n = 4), and quasi-experimental (n = 1). A positive influence of drug dispensing on health outcomes was demonstrated through six clinical, four humanistic and three economic outcomes. Eight studies (88,9 %) used intermediate outcomes. The assessment of methodological quality was characterized by a lack of clarity and/or lack of information in primary studies.

CONCLUSIONS

Most articles included in this review reported a positive influence of drug dispensing performed by community pharmacists on patients' health outcomes. The findings of this study may be of interest to patients, pharmacists, decision makers, and healthcare systems, since they may contribute to evidence-based decision-making, strengthening the contribution of community pharmacists to health care.

TRIAL REGISTRATION

Registration: PROSPERO CRD42020191701 .

Statements considering intervention effects in Finnish clinical practice guidelines: Recommending interventions with non-numeric effect-sizes or unspecified outcomes

RATIONALE, AIMS AND OBJECTIVES

Representation of benefits and harms associated with specific interventions in an understandable and comparable way is crucial for informed decision making that clinical practice guidelines (CPGs) aim to enhance. Therefore, we investigated how statements concerning the effects of interventions considered and described benefits and harms, magnitude of effect and its uncertainty, numeric and non-numeric information, and outcomes in Finnish CPGs.

METHODS

We selected 10 CPGs on common diseases and risk factors published by The Finnish Medical Society, Duodecim. All the statements which were graded with the level of evidence from high to very low (levels A-D) were included in analyses. From these statements, assessments were made regarding whether the statement considered benefits or harms, whether relative or absolute numeric measures were shown, whether the statement supported or was against the intervention considered, and what outcome was reported.

RESULTS

Of the 10 CPGs, 448 statements were assessed. Most of the statements of effects considered intervention benefits (87%) rather than harms. Half of the statements considering harms were represented in a way that supported the intervention. Most of the statements (94%) did not include numeric estimates of magnitude of the effect. When numeric estimates of magnitude of the effect were present, they were most frequently relative measures and were typically placed in a statement considering (a) intervention benefits with a primary outcome, (b) given the grade of A for level of evidence, and (c) that supported the use of intervention.

CONCLUSIONS

In the Finnish CPGs, the statements were rarely framed with both absolute and relative numeric measures of an intervention's effect. Harms were rarely reported with a grade indicating the level of evidence. The users of CPGs would benefit from more consistent and understandable framing of statements considering both benefits and harms of interventions.