1
|
An evaluation of DistillerSR's machine learning-based prioritization tool for title/abstract screening - impact on reviewer-relevant outcomes. BMC Med Res Methodol 2020; 20:256. [PMID: 33059590 PMCID: PMC7559198 DOI: 10.1186/s12874-020-01129-1] [Citation(s) in RCA: 36] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2020] [Accepted: 09/22/2020] [Indexed: 01/14/2023] Open
Abstract
BACKGROUND Systematic reviews often require substantial resources, partially due to the large number of records identified during searching. Although artificial intelligence may not be ready to fully replace human reviewers, it may accelerate and reduce the screening burden. Using DistillerSR (May 2020 release), we evaluated the performance of the prioritization simulation tool to determine the reduction in screening burden and time savings. METHODS Using a true recall @ 95%, response sets from 10 completed systematic reviews were used to evaluate: (i) the reduction of screening burden; (ii) the accuracy of the prioritization algorithm; and (iii) the hours saved when a modified screening approach was implemented. To account for variation in the simulations, and to introduce randomness (through shuffling the references), 10 simulations were run for each review. Means, standard deviations, medians and interquartile ranges (IQR) are presented. RESULTS Among the 10 systematic reviews, using true recall @ 95% there was a median reduction in screening burden of 47.1% (IQR: 37.5 to 58.0%). A median of 41.2% (IQR: 33.4 to 46.9%) of the excluded records needed to be screened to achieve true recall @ 95%. The median title/abstract screening hours saved using a modified screening approach at a true recall @ 95% was 29.8 h (IQR: 28.1 to 74.7 h). This was increased to a median of 36 h (IQR: 32.2 to 79.7 h) when considering the time saved not retrieving and screening full texts of the remaining 5% of records not yet identified as included at title/abstract. Among the 100 simulations (10 simulations per review), none of these 5% of records were a final included study in the systematic review. The reduction in screening burden to achieve true recall @ 95% compared to @ 100% resulted in a reduced screening burden median of 40.6% (IQR: 38.3 to 54.2%). CONCLUSIONS The prioritization tool in DistillerSR can reduce screening burden. A modified or stop screening approach once a true recall @ 95% is achieved appears to be a valid method for rapid reviews, and perhaps systematic reviews. This needs to be further evaluated in prospective reviews using the estimated recall.
Collapse
|
2
|
Ahmadzai N, Cheng W, Kilty S, Esmaeilisaraji L, Wolfe D, Bonaparte J, Schramm D, Fitzpatrick E, Lin V, Skidmore B, Hutton B. Pharmacologic and surgical therapies for patients with Meniere's disease: A systematic review and network meta-analysis. PLoS One 2020; 15:e0237523. [PMID: 32870918 PMCID: PMC7462264 DOI: 10.1371/journal.pone.0237523] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2020] [Accepted: 07/28/2020] [Indexed: 11/28/2022] Open
Abstract
BACKGROUND Meniere's disease (MD) is a chronic condition of the inner ear consisting of symptoms that include vertigo attacks, fluctuating sensorineural hearing loss, tinnitus and aural fullness. Despite availability of various interventions, there is uncertainty surrounding their relative efficacy, thus making it difficult to select the appropriate treatments for MD. The objective of this systematic review was to assess the relative effects of the available pharmacologic and surgical interventions in patients with MD with regard to vertigo and other key patient outcomes based on data from randomized clinical trials (RCTs). METHODS Our published protocol registered with PROSPERO (CRD42019119129) provides details on eligibility criteria and methods. We searched various databases including MEDLINE, Embase and the Cochrane Library from inception to December 10th, 2018. Screening at citation and full-text levels and risk of bias assessment were performed by two independent reviewers in duplicate, with discrepancies resolved by consensus or third-party adjudication. Bayesian network meta-analyses (NMA) were performed for hearing change and vertigo control outcomes, along with pairwise meta-analyses for these and additional outcomes. RESULTS We identified 2,889 unique citations, that yielded 23 relevant publications describing 18 unique RCTs (n = 1,231 patients). Overall, risk-of bias appraisal suggested the evidence base to be at unclear or high risk of bias. Amongst pharmacologics, we constructed treatment networks of five intervention groups that included placebo, intratympanic (IT) gentamicin, oral high-dose betahistine, IT steroid and IT steroid plus high-dose betahistine for NMAs of hearing change (improvement or deterioration) and complete vertigo control. IT steroid plus high-dose betahistine was associated with the largest difference in hearing improvement compared to placebo, followed by high-dose betahistine and IT steroid (though 95% credible intervals failed to rule out the possibility of no difference), while IT gentamicin was worse than IT steroid. The NMA of complete vertigo control suggested IT gentamicin was associated with the highest probability of achieving better complete vertigo control compared to placebo, followed by IT steroid plus high-dose betahistine. Only two studies related to surgical interventions were found, and data suggested no statistically significant difference in hearing changes between endolymphatic duct blockage (EDB) versus endolymphatic sac decompression (ESD), and ESD with or without steroid injection. One trial reported that 96.5% of patients in EDB group compared to 37.5% of the patients in ESD group achieved complete vertigo control 24 months after surgery (p = 0.002). CONCLUSION To achieve both hearing preservation and vertigo control, the best treatment option among the pharmacologic interventions compared may be IT steroid plus high-dose betahistine, considering that IT gentamicin may have good performance to control vertigo but may be detrimental to hearing preservation with high cumulative dosage and short interval between injections. However, IT steroid plus high-dose betahistine has not been compared in head-to-head trials against other interventions except for IT steroid alone in one trial, thus future trials that compare it with other interventions will help establish comparative effectiveness with direct evidence.
Collapse
Affiliation(s)
- Nadera Ahmadzai
- Ottawa Hospital Research Institute, Clinical Epidemiology Program, Ottawa, Canada
| | - Wei Cheng
- Ottawa Hospital Research Institute, Clinical Epidemiology Program, Ottawa, Canada
| | - Shaun Kilty
- Ottawa Hospital Research Institute, Clinical Epidemiology Program, Ottawa, Canada
- Department of Otolaryngology – Head and Neck Surgery, The Ottawa Hospital, Ottawa, Canada
- Dr. S. Kilty Medicine Prof. Corp, Ottawa, Canada
| | - Leila Esmaeilisaraji
- Ottawa Hospital Research Institute, Clinical Epidemiology Program, Ottawa, Canada
| | - Dianna Wolfe
- Ottawa Hospital Research Institute, Clinical Epidemiology Program, Ottawa, Canada
| | - James Bonaparte
- Department of Otolaryngology – Head and Neck Surgery, The Ottawa Hospital, Ottawa, Canada
| | - David Schramm
- Ottawa Hospital Research Institute, Clinical Epidemiology Program, Ottawa, Canada
- Department of Otolaryngology – Head and Neck Surgery, The Ottawa Hospital, Ottawa, Canada
| | - Elizabeth Fitzpatrick
- The University of Ottawa Faculty of Epidemiology and Community Medicine, Ottawa, Canada
- Faculty of Health Sciences, University of Ottawa, Ottawa, Canada
| | - Vincent Lin
- CHEO Research Institute, Ottawa, Canada
- Department of Otolaryngology – Head & Neck Surgery, Sunnybrook Health Sciences Centre, Sunnybrook Research Institute, Toronto, Canada
| | - Becky Skidmore
- Ottawa Hospital Research Institute, Clinical Epidemiology Program, Ottawa, Canada
| | - Brian Hutton
- Ottawa Hospital Research Institute, Clinical Epidemiology Program, Ottawa, Canada
- Faculty of Medicine, University of Toronto, Toronto, Canada
| |
Collapse
|
3
|
Adile AA, Kameda-Smith MM, Bakhshinyan D, Banfield L, Salim SK, Farrokhyar F, Fleming AJ. Salvage therapy for progressive, treatment-refractory or recurrent pediatric medulloblastoma: a systematic review protocol. Syst Rev 2020; 9:47. [PMID: 32127049 PMCID: PMC7055028 DOI: 10.1186/s13643-020-01307-8] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/30/2019] [Accepted: 02/20/2020] [Indexed: 12/17/2022] Open
Abstract
BACKGROUND Central nervous system tumors remain the leading cause of cancer-related mortality amongst children with solid tumors, with medulloblastoma (MB) representing the most common pediatric brain malignancy. Despite best current therapies, patients with recurrent MB experience have an alarmingly high mortality rate and often have limited therapeutic options beyond inadequate chemotherapy or experimental clinical trials. Therefore, a systematic review of the literature regarding treatment strategies employed in recurrent pediatric MB will evaluate previous salvage therapies in order to guide future clinical trials. The aim of this systematic review will be to investigate the efficacy and safety of salvage therapies for the management of children with progressive, treatment-refractory, or recurrent MB. METHODS We will conduct literature searches (from 1995 onwards) in MEDLINE, EMBASE, ClinicalTrials.gov, WHO International Clinical Trials Registry Platform, and Cochrane Central Register of Controlled Trials. Studies examining the survival and toxicity of therapies administered to treatment-refractory pediatric MB patients will be included. Two reviewers will independently assess the search results based on predefined selection criteria, complete data abstraction, and quality assessment. The primary outcomes of this review will be overall and progression-free survival. Secondary outcomes will include safety and toxicity of each therapy administered. The study methodological quality (or bias) will be appraised using an appropriate tool. Due to the nature of the research question and published literature, we expect large inter-study heterogeneity and therefore will use random effects regression analysis to extract the combined effect. In additional analyses, we will investigate the role of re-irradiation and mono- vs. poly-therapy in recurrent disease, and whether molecular subgrouping of MB influences salvage therapy. DISCUSSION This systematic review will provide an overview of the current literature regarding salvage therapies for relapsed MB patients. Investigation of clinically tested therapies for children with recurrent MB has significant implications for clinical practice. By reviewing the efficacy and toxicity of MB salvage therapies, this study will identify effective therapeutic strategies administered to recurrent MB patients and can inform future clinical trials aimed to improve patient survivorship and quality of life. SYSTEMATIC REVIEW REGISTRATION PROSPERO CRD42020167421.
Collapse
Affiliation(s)
- Ashley A. Adile
- McMaster Stem Cell and Cancer Research Institute, McMaster University, Hamilton, ON Canada
- Department of Biochemistry and Biomedical Sciences, McMaster University, Hamilton, ON Canada
| | - Michelle M. Kameda-Smith
- McMaster Stem Cell and Cancer Research Institute, McMaster University, Hamilton, ON Canada
- Department of Surgery, Division of Neurosurgery, McMaster University, Hamilton, ON Canada
| | - David Bakhshinyan
- McMaster Stem Cell and Cancer Research Institute, McMaster University, Hamilton, ON Canada
- Department of Biochemistry and Biomedical Sciences, McMaster University, Hamilton, ON Canada
| | - Laura Banfield
- Health Sciences Library, McMaster University, Hamilton, Ontario Canada
| | - Sabra K. Salim
- McMaster Stem Cell and Cancer Research Institute, McMaster University, Hamilton, ON Canada
- Department of Biochemistry and Biomedical Sciences, McMaster University, Hamilton, ON Canada
| | - Forough Farrokhyar
- Department of Surgery, Division of Neurosurgery, McMaster University, Hamilton, ON Canada
- Department of Health, Evidence and Impact, McMaster University, Hamilton, Ontario Canada
| | - Adam J. Fleming
- Department of Pediatrics, Division of Hematology-Oncology, McMaster University, Hamilton, Canada
| |
Collapse
|