Study filters for non-randomized studies of interventions consistently lacked sensitivity upon external validation

Evidence from clinical trials on high-risk medical devices in children: a scoping review

BACKGROUND

Meeting increased regulatory requirements for clinical evaluation of medical devices marketed in Europe in accordance with the Medical Device Regulation (EU 2017/745) is challenging, particularly for high-risk devices used in children.

METHODS

Within the CORE-MD project, we performed a scoping review on evidence from clinical trials investigating high-risk paediatric medical devices used in paediatric cardiology, diabetology, orthopaedics and surgery, in patients aged 0-21 years. We searched Medline and Embase from 1st January 2017 to 9th November 2022.

RESULTS

From 1692 records screened, 99 trials were included. Most were multicentre studies performed in North America and Europe that mainly had evaluated medical devices from the specialty of diabetology. Most had enrolled adolescents and 39% of trials included both children and adults. Randomized controlled trials accounted for 38% of the sample. Other frequently used designs were before-after studies (21%) and crossover trials (20%). Included trials were mainly small, with a sample size <100 participants in 64% of the studies. Most frequently assessed outcomes were efficacy and effectiveness as well as safety.

CONCLUSION

Within the assessed sample, clinical trials on high-risk medical devices in children were of various designs, often lacked a concurrent control group, and recruited few infants and young children.

IMPACT

In the assessed sample, clinical trials on high-risk medical devices in children were mainly small, with variable study designs (often without concurrent control), and they mostly enrolled adolescents. We provide a systematic summary of methodologies applied in clinical trials of medical devices in the paediatric population, reflecting obstacles in this research area that make it challenging to conduct adequately powered randomized controlled trials. In view of changing European regulations and related concerns about shortages of high-risk medical devices for children, our findings may assist competent authorities in setting realistic requirements for the evidence level to support device conformity certification.

Ablative and non-surgical therapies for early and very early hepatocellular carcinoma: a systematic review and network meta-analysis

Background

A wide range of ablative and non-surgical therapies are available for treating small hepatocellular carcinoma in patients with very early or early-stage disease and preserved liver function.

Objective

To review and compare the effectiveness of all current ablative and non-surgical therapies for patients with small hepatocellular carcinoma (≤ 3 cm).

Design

Systematic review and network meta-analysis.

Data sources

Nine databases (March 2021), two trial registries (April 2021) and reference lists of relevant systematic reviews.

Review methods

Eligible studies were randomised controlled trials of ablative and non-surgical therapies, versus any comparator, for small hepatocellular carcinoma. Randomised controlled trials were quality assessed using the Cochrane Risk of Bias 2 tool and mapped. The comparative effectiveness of therapies was assessed using network meta-analysis. A threshold analysis was used to identify which comparisons were sensitive to potential changes in the evidence. Where comparisons based on randomised controlled trial evidence were not robust or no randomised controlled trials were identified, a targeted systematic review of non-randomised, prospective comparative studies provided additional data for repeat network meta-analysis and threshold analysis. The feasibility of undertaking economic modelling was explored. A workshop with patients and clinicians was held to discuss the findings and identify key priorities for future research.

Results

Thirty-seven randomised controlled trials (with over 3700 relevant patients) were included in the review. The majority were conducted in China or Japan and most had a high risk of bias or some risk of bias concerns. The results of the network meta-analysis were uncertain for most comparisons. There was evidence that percutaneous ethanol injection is inferior to radiofrequency ablation for overall survival (hazard ratio 1.45, 95% credible interval 1.16 to 1.82), progression-free survival (hazard ratio 1.36, 95% credible interval 1.11 to 1.67), overall recurrence (relative risk 1.19, 95% credible interval 1.02 to 1.39) and local recurrence (relative risk 1.80, 95% credible interval 1.19 to 2.71). Percutaneous acid injection was also inferior to radiofrequency ablation for progression-free survival (hazard ratio 1.63, 95% credible interval 1.05 to 2.51). Threshold analysis showed that further evidence could plausibly change the result for some comparisons. Fourteen eligible non-randomised studies were identified (n ≥ 2316); twelve had a high risk of bias so were not included in updated network meta-analyses. Additional non-randomised data, made available by a clinical advisor, were also included (n = 303). There remained a high level of uncertainty in treatment rankings after the network meta-analyses were updated. However, the updated analyses suggested that microwave ablation and resection are superior to percutaneous ethanol injection and percutaneous acid injection for some outcomes. Further research on stereotactic ablative radiotherapy was recommended at the workshop, although it is only appropriate for certain patient subgroups, limiting opportunities for adequately powered trials.

Limitations

Many studies were small and of poor quality. No comparative studies were found for some therapies.

Conclusions

The existing evidence base has limitations; the uptake of specific ablative therapies in the United Kingdom appears to be based more on technological advancements and ease of use than strong evidence of clinical effectiveness. However, there is evidence that percutaneous ethanol injection and percutaneous acid injection are inferior to radiofrequency ablation, microwave ablation and resection.

Study registration

PROSPERO CRD42020221357.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment (HTA) programme (NIHR award ref: NIHR131224) and is published in full in Health Technology Assessment; Vol. 27, No. 29. See the NIHR Funding and Awards website for further award information.

Anxiety, depression, and sleep-related outcomes of glaucoma patients: systematic review and meta-analysis

OBJECTIVE

This study aims to synthesize data quantifying the prevalence and severity of common psychological conditions in patients with glaucoma.

DESIGN

Systematic review and meta-analysis.

METHODS

Databases including Ovid MEDLINE, CINAHL, EMBASE, PsycINFO, Cochrane Library, Web of Science, Open Grey, and ProQuest Theses and dissertations were searched. Two reviewers independently assessed and screened all studies, followed by quality assessment of included studies using the modified Downs and Black checklist. Data were pooled using fixed-effect and random-effects models.

RESULTS

Of 2067 studies identified by the search strategy, 57 passed full-text screening, and 45 studies (4 995 538 subjects) were eligible for analysis. Overall, the prevalence of depression (effect size [ES] = 0.19, 95% CI 0.16-0.23; n = 31), anxiety (ES = 0.25, 95% CI 0.21-0.30; n = 18), and sleep disorders (ES = 0.47, 95% CI 0.26-0.68; n = 7) were high in patients with glaucoma. Similarly, symptomatic measurements of depression (standardized mean difference [SMD] = 0.46, 95% CI 0.19-0.73), anxiety (SMD = 0.44, 95% CI 0.08-0.81), and sleep quality (SMD = 0.72, 95% CI 0.22-1.21) were significant in glaucoma patients.

CONCLUSIONS

A higher prevalence and severity of depression, anxiety, and sleep disorders was experienced in patients with glaucoma compared with patients without glaucoma. Caregivers as well as health care providers may need to be aware of unique psychological and social stressors placed on glaucoma patients.

Home-built environment interventions and inflammation biomarkers: a systematic review and meta-analysis protocol

BACKGROUND

Inflammation control is a fundamental part of chronic care in patients with a history of cancer and comorbidity. As the risk-benefit profile of anti-inflammatory drugs is unclear in survivors of cancer, GPs and patients could benefit from alternative non-pharmacological treatment options for dysregulated inflammation. There is a potential for home-built environment (H-BE) interventions to modulate inflammation; however, discrepancies exist between studies.

AIM

To evaluate the effectiveness of H-BE interventions on cancer-associated inflammation biomarkers.

DESIGN & SETTING

A systematic review and meta-analysis of randomised and non-randomised trials in community-dwelling adults.

METHOD

PubMed and MEDLINE, Embase, Web of Science, and Google Scholar will be searched for clinical trials published in January 2000 onwards. The study will include H-BE interventions modifying air quality, thermal comfort, non-ionising radiation, noise, nature, and water. No restrictions to study population will be applied to allow deriving expectations for effects of the interventions in cancer survivors from available source populations. Outcome measures will be inflammatory biomarkers clinically and physiologically relevant to cancer. The first reviewer will independently screen articles together with GPs and extract data that will be verified by a second reviewer. The quality of studies will be assessed using the Cochrane risk-of-bias tools. Depending on the clinical and methodological homogeneity of populations, interventions, and outcomes, a meta-analysis will be conducted using random-effects models.

CONCLUSION

Findings will determine the effectiveness of H-BE interventions on inflammatory parameters, guide future directions for its provision in community-dwelling survivors of cancer and support GPs with safer anti-inflammatory treatment options in high-risk patients for clinical complications.

Applying machine classifiers to update searches: Analysis from two case studies

Manual screening of citation records could be reduced by using machine classifiers to remove records of very low relevance. This seems particularly feasible for update searches, where a machine classifier can be trained from past screening decisions. However, feasibility is unclear for broad topics. We evaluate the performance and implementation of machine classifiers for update searches of public health research using two case studies. The first study evaluates the impact of using different sets of training data on classifier performance, comparing recall and screening reduction with a manual screening ‘gold standard’. The second study uses screening decisions from a review to train a classifier that is applied to rank the update search results. A stopping threshold was applied in the absence of a gold standard. Time spent screening titles and abstracts of different relevancy‐ranked records was measured. Results: Study one: Classifier performance varies according to the training data used; all custom‐built classifiers had a recall above 93% at the same threshold, achieving screening reductions between 41% and 74%. Study two: applying a classifier provided a solution for tackling a large volume of search results from the update search, and screening volume was reduced by 61%. A tentative estimate indicates over 25 h screening time was saved. In conclusion, custom‐built machine classifiers are feasible for reducing screening workload from update searches across a range of public health interventions, with some limitation on recall. Key considerations include selecting a training dataset, agreeing stopping thresholds and processes to ensure smooth workflows.

Individual-level interventions to reduce personal exposure to outdoor air pollution and their effects on people with long-term respiratory conditions

BACKGROUND

More than 90% of the global population lives in areas exceeding World Health Organization air quality limits. More than four million people each year are thought to die early due to air pollution, and poor air quality is thought to reduce an average European's life expectancy by one year. Individuals may be able to reduce health risks through interventions such as masks, behavioural changes and use of air quality alerts. To date, evidence is lacking about the efficacy and safety of such interventions for the general population and people with long-term respiratory conditions. This topic, and the review question relating to supporting evidence to avoid or lessen the effects of air pollution, emerged directly from a group of people with chronic obstructive pulmonary disease (COPD) in South London, UK.

OBJECTIVES

1. To assess the efficacy, safety and acceptability of individual-level interventions that aim to help people with or without chronic respiratory conditions to reduce their exposure to outdoor air pollution. 2. To assess the efficacy, safety and acceptability of individual-level interventions that aim to help people with chronic respiratory conditions reduce the personal impact of outdoor air pollution and improve health outcomes.

SEARCH METHODS

We identified studies from the Cochrane Airways Trials Register, Cochrane Central Register of Controlled Trials, and other major databases. We did not restrict our searches by date, language or publication type and included a search of the grey literature (e.g. unpublished information). We conducted the most recent search on 16 October 2020.

SELECTION CRITERIA

We included randomised controlled trials (RCTs) and non-randomised studies (NRS) that included a comparison treatment arm, in adults and children that investigated the effectiveness of an individual-level intervention to reduce risks of outdoor air pollution. We included studies in healthy individuals and those in people with long-term respiratory conditions. We excluded studies which focused on non-respiratory long-term conditions, such as cardiovascular disease. We did not restrict eligibility of studies based on outcomes.

DATA COLLECTION AND ANALYSIS

We used standard Cochrane methods. Two review authors independently selected trials for inclusion, extracted study characteristics and outcome data, and assessed risk of bias using the Cochrane Risk of Bias tool for RCTs and the Risk Of Bias In Non-randomised Studies - of Interventions (ROBINS-I) as appropriate. One review author entered data into the review; this was spot-checked by a second author. We planned to meta-analyse results from RCTs and NRS separately, using a random-effects model. This was not possible, so we presented evidence narratively. We assessed certainty of the evidence using the GRADE approach. Primary outcomes were: measures of air pollution exposure; exacerbation of respiratory conditions; hospital admissions; quality of life; and serious adverse events.

MAIN RESULTS

We identified 11 studies (3372 participants) meeting our inclusion criteria (10 RCTs and one NRS). Participants' ages ranged from 18 to 74 years, and the duration of studies ranged from 24 hours to 104 weeks. Six cross-over studies recruited healthy adults and five parallel studies included either people with pre-existing conditions (three studies) or only pregnant women (two studies). Interventions included masks (e.g. an N95 mask designed to filter out airborne particles) (five studies), an alternative cycle route (one study), air quality alerts and education (five studies). Studies were set in Australia, China, Iran, the UK, and the USA. Due to the diversity of study designs, populations, interventions and outcomes, we did not perform any meta-analyses and instead summarised results narratively. We judged both RCTs and the NRS to be at risk of bias from lack of blinding and lack of clarity regarding selection methods. Many studies did not provide a prepublished protocol or trial registration. From five studies (184 participants), we found that masks or altered cycle routes may have little or no impact on physiological markers of air pollution exposure (e.g. blood pressure and heart rate variability), but we are very uncertain about this estimate using the GRADE approach. We found conflicting evidence regarding health care usage from three studies of air pollution alerts, with one non-randomised cross-over trial (35 participants) reporting an increase in emergency hospital attendances and admissions, but the other two randomised parallel trials (1553 participants) reporting little to no difference. We also gave the evidence for this outcome a very uncertain GRADE rating. None of our included trials reported respiratory exacerbations, quality of life or serious adverse events. Secondary outcomes were not well reported, but indicated inconsistent impacts of air quality alerts and education interventions on adherence, with some trials reporting improvements in the intervention groups and others reporting little or no difference. Symptoms were reported by three trials, with one randomised cross-over trial (15 participants) reporting a small increase in breathing difficulties associated with the mask intervention, one non-randomised cross-over trial (35 participants) reporting reduced throat and nasal irritation in the lower-pollution cycle route group (but no clear difference in other respiratory symptoms), and another randomised parallel trial (519 participants) reporting no clear difference in symptoms between those who received a smog warning and those who did not.

AUTHORS' CONCLUSIONS

The lack of evidence and study diversity has limited the conclusions of this review. Using a mask or a lower-pollution cycle route may mitigate some of the physiological impacts from air pollution, but evidence was very uncertain. We found conflicting results for other outcomes, including health care usage, symptoms and adherence/behaviour change. We did not find evidence for adverse events. Funders should consider commissioning larger, longer studies, using high-quality and well-described methods, recruiting participants with pre-existing respiratory conditions. Studies should report outcomes of importance to people with respiratory conditions, such as exacerbations, hospital admissions, quality of life and adverse events.

Publication Trends of Pediatric and Adult Randomized Controlled Trials in General Medical Journals, 2005-2018: A Citation Analysis

Policy has been developed to promote the conduct of high-quality pediatric randomized controlled trials (RCTs). Whether these strategies have influenced publication trends in high-impact journals is unknown. We aim to evaluate characteristics, citation patterns, and publication trends of pediatric RCTs published in general medical journals (GMJs) compared with adult RCTs over a 13-year period. Studies were identified using Medline, and impact metrics were collected from Web of Science and Scopus. All RCTs published from 2005–2018 in 7 GMJs with the highest impact factors were identified for analysis. A random sample of matched pediatric and adult RCTs were assessed for publication characteristics, academic and non-academic citation. Citations were counted from publication until June 2019. Among 4146 RCTs, 2794 (67.3%) enrolled adults, 591 (14.2%) enrolled children, and 761 RCTs (18.3%) enrolled adult and pediatric patients. Adult RCTs published in GMJs grew by 5.1 publications per year (95% CI: 3.3–6.9), while the number of pediatric RCTs did not show significant change (−0.4 RCTs/year, 95% CI: −1.4–0.6). Adult RCTs were cited more than pediatric RCTs (median(IQR): 29.9 (68.5–462.8) citations/year vs. 13.2 (6.8–24.9) citations/year; p < 0.001); however, social media attention was similar (median(IQR) Altmetric Attention Score: 37 (13.75–133.8) vs. 26 (6.2–107.5); p = 0.25). Despite policies which may facilitate conduct of pediatric RCTs, the publishing gap in high-impact GMJs is widening.

Development and validation of study filters for identifying controlled non-randomized studies in PubMed and Ovid MEDLINE

A retrospective analysis published by the German Institute for Quality and Efficiency in Health Care (IQWiG) in 2018 concluded that no filter for non-randomized studies (NRS) achieved sufficient sensitivity (≥92%), a precondition for comprehensive information retrieval. New NRS filters are therefore required, taking into account the challenges related to this study type. Our evaluation focused on the development of study filters for NRS with a control group ("controlled NRS"), as this study type allows the calculation of an effect size. In addition, we assumed that due to the more explicit search syntax, controlled NRS are easier to identify than non-controlled ones, potentially resulting in better performance measures of study filters for controlled NRS. Our aim was to develop study filters for identifying controlled NRS in PubMed and Ovid MEDLINE. We developed two new search filters that can assist clinicians and researchers in identifying controlled NRS in PubMed and Ovid MEDLINE. The reference set was based on 2110 publications in Medline extracted from 271 Cochrane reviews and on 4333 irrelevant references. The first filter maximizes sensitivity (92.42%; specificity 79.67%, precision 68.49%) and should be used when a comprehensive search is needed. The second filter maximizes specificity (92.06%; precision 82.98%, sensitivity 80.94%) and should be used when a more focused search is sufficient.