CHARMS and PROBAST at your fingertips: a template for data extraction and risk of bias assessment in systematic reviews of predictive models

Opioids and personalized analgesia in the perioperative setting: A protocol for five systematic reviews

BACKGROUND

Treatment with opioids is a mainstay in perioperative pain management. While the leading treatment paradigm has been procedure-specific pain management, efforts regarding personalized pain treatment are increasing. The OPI•AID project aims to develop personalized algorithms for perioperative pain management, taking demographic, surgical, and anaesthesiologic factors into account. We will undertake five parallel reviews to illuminate current evidence on different aspects of individual responses to perioperative opioid treatment.

METHODS

Inclusion of adult populations in English-written studies. Review-specific searches are developed for the following databases: CENTRAL, MEDLINE, Embase, clinicaltrials.gov, and clinicaltrial.eu. Two authors will independently screen citations, extract data, and assess the risks of bias in each review (QUIPS, PROBAST and RoB2, as relevant).

CONCLUSION

These reviews will evaluate various aspects of perioperative opioid treatment, including individualized treatment strategies, selection of specific opioids, and individual patient responses. These will guide future development of a personalized perioperative opioid treatment algorithm (OPI•AID) that will be validated and tested clinically against standard of care.

Post-Cardiac arrest outcome prediction using machine learning: A systematic review and meta-analysis

BACKGROUND

Early and reliable prognostication in post-cardiac arrest patients remains challenging, with various factors linked to return of spontaneous circulation (ROSC), survival, and neurological results. Machine learning and deep learning models show promise in improving these predictions. This systematic review and meta-analysis evaluates how effective these approaches are in predicting clinical outcomes at different time points using structured data.

METHODS

This study followed PRISMA guidelines, involving a comprehensive search across PubMed, Scopus, and Web of Science databases until March 2024. Studies aimed at predicting ROSC, survival (or mortality), and neurological outcomes after cardiac arrest through the application of machine learning or deep learning techniques with structured data were included. Data extraction followed the guidelines of the CHARMS checklist, and the bias risk was evaluated using PROBAST tool. Models reporting the AUC metric with 95 % confidence intervals were incorporated into the quantitative synthesis and meta-analysis.

RESULTS

After extracting 2,753 initial records, 41 studies met the inclusion criteria, yielding 97 machine learning and 16 deep learning models. The pooled AUC for predicting favorable neurological outcomes (CPC 1 or 2) at hospital discharge was 0.871 (95 % CI: 0.813 - 0.928) for machine learning models and 0.877 (95 % CI: 0.831-0.924) across deep learning algorithms. For survival prediction, this value was found to be 0.837 (95 % CI: 0.757-0.916). Considerable heterogeneity and high risk of bias were observed, mainly attributable to inadequate management of missing data and the absence of calibration plots. Most studies focused on pre-hospital factors, with age, sex, and initial arrest rhythm being the most frequent features.

CONCLUSION

Predictive models utilizing AI-based approaches, including machine and deep learning models exhibit enhanced effectiveness compared to previous regression algorithms, but significant heterogeneity and high risk of bias limit their dependability. Evaluating state-of-the-art deep learning models tailored for tabular data and their clinical generalizability can enhance outcome prediction after cardiac arrest.

Targeted Development and Validation of Clinical Prediction Models in Secondary Care Settings: Opportunities and Challenges for Electronic Health Record Data

Unlabelled

Before deploying a clinical prediction model (CPM) in clinical practice, its performance needs to be demonstrated in the population of intended use. This is also called "targeted validation." Many CPMs developed in tertiary settings may be most useful in secondary care, where the patient case mix is broad and practitioners need to triage patients efficiently. However, since structured or rich datasets of sufficient quality from secondary to assess the performance of a CPM are scarce, a validation gap exists that hampers the implementation of CPMs in secondary care settings. In this viewpoint, we highlight the importance of targeted validation and the use of CPMs in secondary care settings and discuss the potential and challenges of using electronic health record (EHR) data to overcome the existing validation gap. The introduction of software applications for text mining of EHRs allows the generation of structured "big" datasets, but the imperfection of EHRs as a research database requires careful validation of data quality. When using EHR data for the development and validation of CPMs, in addition to widely accepted checklists, we propose considering three additional practical steps: (1) involve a local EHR expert (clinician or nurse) in the data extraction process, (2) perform validity checks on the generated datasets, and (3) provide metadata on how variables were constructed from EHRs. These steps help to generate EHR datasets that are statistically powerful, of sufficient quality and replicable, and enable targeted development and validation of CPMs in secondary care settings. This approach can fill a major gap in prediction modeling research and appropriately advance CPMs into clinical practice.

Artificial Intelligence for Prediction of Shunt Response in Idiopathic Normal Pressure Hydrocephalus: A Systematic Review

BACKGROUND

Idiopathic normal pressure hydrocephalus (iNPH) is a reversible cause of dementia, typically treated with shunt surgery, although outcomes vary. Artificial intelligence (AI) advancements could improve predictions of shunt response (SR) by analyzing extensive datasets.

METHODS

We conducted a systematic review to assess AI's effectiveness in predicting SR in iNPH. Studies using AI or machine learning algorithms for SR prediction were identified through searches in MEDLINE, Embase, and Web of Science up to September 2023, adhering to Synthesis Without Meta-Analysis reporting guidelines.

RESULTS

Of 3541 studies identified, 33 were assessed for eligibility, and 8 involving 479 patients were included. Study sample sizes varied from 28 to 132 patients. Common data inputs included imaging/radiomics (62.5%) and demographics (37.5%), with Support Vector Machine being the most frequently used machine learning algorithm (87.5%). Two studies compared multiple algorithms. Only 4 studies reported the Area Under the Curve values, which ranged between 0.80 and 0.94. The results highlighted inconsistency in outcome measures, data heterogeneity, and potential biases in the models used.

CONCLUSIONS

While AI shows promise for improving iNPH management, there is a need for standardized data and extensive validation of AI models to enhance their clinical utility. Future research should aim to develop robust and generalizable AI models for more effective diagnosis and management of iNPH.

Prediction Models for Perioperative Blood Transfusion in Patients Undergoing Gynecologic Surgery: A Systematic Review

This systematic review aimed to evaluate prediction models for perioperative blood transfusion in patients undergoing gynecologic surgery. Given the inherent risks associated with blood transfusion and the critical need for accurate prediction, this study identified and assessed models based on their development, validation, and predictive performance. The review included five studies encompassing various surgical procedures and approaches. Predicting factors commonly used across these models included preoperative hematocrit, race, surgical route, and uterine fibroid characteristics. However, the review highlighted significant variability in the definition of perioperative periods, a lack of standardization in transfusion criteria, and a high risk of bias in most models due to methodological issues, such as a low number of events per variable, inappropriate handling of continuous and categorical predictors, inappropriate handling of missing data, improper methods of predictor selection, inappropriate measurement methods for model performance, and inadequate evaluations of model overfitting and optimism in model performance. Despite some models demonstrating good discrimination and calibration, the overall quality and external validation of these models were limited. Consequently, there is a clear need for more robust and externally validated models to improve clinical decision-making and patient outcomes in gynecologic surgery. Future research should focus on refining these models, incorporating rigorous validation, and adhering to standardized reporting practices.

Diagnostic accuracy of deep learning in detection and prognostication of renal cell carcinoma: a systematic review and meta-analysis

Introduction

The prevalence of Renal cell carcinoma (RCC) is increasing among adults. Histopathologic samples obtained after surgical resection or from biopsies of a renal mass require subtype classification for diagnosis, prognosis, and to determine surveillance. Deep learning in artificial intelligence (AI) and pathomics are rapidly advancing, leading to numerous applications such as histopathological diagnosis. In our meta-analysis, we assessed the pooled diagnostic performances of deep neural network (DNN) frameworks in detecting RCC subtypes and to predicting survival.

Methods

A systematic search was done in PubMed, Google Scholar, Embase, and Scopus from inception to November 2023. The random effects model was used to calculate the pooled percentages, mean, and 95% confidence interval. Accuracy was defined as the number of cases identified by AI out of the total number of cases, i.e. (True Positive + True Negative)/(True Positive + True Negative + False Positive + False Negative). The heterogeneity between study-specific estimates was assessed by the I 2 statistic. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were used to conduct and report the analysis.

Results

The search retrieved 347 studies; 13 retrospective studies evaluating 5340 patients were included in the final analysis. The pooled performance of the DNN was as follows: accuracy 92.3% (95% CI: 85.8-95.9; I 2 = 98.3%), sensitivity 97.5% (95% CI: 83.2-99.7; I 2 = 92%), specificity 89.2% (95% CI: 29.9-99.4; I 2 = 99.6%) and area under the curve 0.91 (95% CI: 0.85-0.97.3; I 2 = 99.6%). Specifically, their accuracy in RCC subtype detection was 93.5% (95% CI: 88.7-96.3; I 2 = 92%), and the accuracy in survival analysis prediction was 81% (95% CI: 67.8-89.6; I 2 = 94.4%).

Discussion

The DNN showed excellent pooled diagnostic accuracy rates to classify RCC into subtypes and grade them for prognostic purposes. Further studies are required to establish generalizability and validate these findings on a larger scale.

Predicting Recovery After Concussion in Pediatric Patients: A Meta-Analysis

CONTEXT

Prognostic prediction models (PPMs) can help clinicians predict outcomes.

OBJECTIVE

To critically examine peer-reviewed PPMs predicting delayed recovery among pediatric patients with concussion.

DATA SOURCES

Ovid Medline, Embase, Ovid PsycInfo, Web of Science Core Collection, Cumulative Index to Nursing and Allied Health Literature, Cochrane Library, Google Scholar.

STUDY SELECTION

The study had to report a PPM for pediatric patients to be used within 28 days of injury to estimate risk of delayed recovery at 28 days to 1 year postinjury. Studies had to have at least 30 participants.

DATA EXTRACTION

The Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies checklist was completed.

RESULTS

Six studies of 13 PPMs were included. These studies primarily reflected male patients in late childhood or early adolescence presenting to an emergency department meeting the Concussion in Sport Group concussion criteria. No study authors used the same outcome definition nor evaluated the clinical utility of a model. All studies demonstrated high risk of bias. Quality of evidence was best for the Predicting and Preventing Postconcussive Problems in Pediatrics (5P) clinical risk score.

LIMITATIONS

No formal PPM Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) process exists.

CONCLUSIONS

The 5P clinical risk score may be considered for clinical use. Rigorous external validations, particularly in other settings, are needed. The remaining PPMs require external validation. Lack of consensus regarding delayed recovery criteria limits these PPMs.

Artificial Intelligence-Assisted MRI Diagnosis in Lumbar Degenerative Disc Disease: A Systematic Review

STUDY DESIGN

Systematic review.

OBJECTIVES

Lumbar degenerative disc disease (DDD) poses a significant global health care challenge, with accurate diagnosis being difficult using conventional methods. Artificial intelligence (AI), particularly machine learning and deep learning, offers promising tools for improving diagnostic accuracy and workflow in lumbar DDD. This study aims to review AI-assisted magnetic resonance imaging (MRI) diagnosis in lumbar DDD and discuss current research for clinical use.

METHODS

A systematic search of electronic databases identified studies on AI applications in MRI-based lumbar DDD diagnosis, following Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines. Search terms included combinations of "Artificial Intelligence," "Machine Learning," "Deep Learning," "Low Back Pain," "Lumbar," "Disc," "Degeneration," and "MRI," targeting studies in English from January 1, 2010, to January 1, 2024. Inclusion criteria encompassed experimental and observational studies in peer-reviewed journals. Data extraction focused on study characteristics, AI techniques, performance metrics, and diagnostic outcomes, with quality assessed using predefined criteria.

RESULTS

Twenty studies met the inclusion criteria, employing various AI methodologies, including machine learning and deep learning, to diagnose lumbar DDD manifestations such as disc degeneration, herniation, and bulging. AI models consistently outperformed conventional methods in accuracy, sensitivity, and specificity, with performance metrics ranging from 71.5% to 99% across different diagnostic objectives.

CONCLUSION

The algorithm model provides a structured framework for integrating AI into routine clinical practice, enhancing diagnostic precision and patient outcomes in lumbar DDD management. Further research and validation are needed to refine AI algorithms for real-world application in lumbar DDD diagnosis.

Machine Learning Models for Pancreatic Cancer Risk Prediction Using Electronic Health Record Data-A Systematic Review and Assessment

INTRODUCTION

Accurate risk prediction can facilitate screening and early detection of pancreatic cancer (PC). We conducted a systematic review to critically evaluate effectiveness of machine learning (ML) and artificial intelligence (AI) techniques applied to electronic health records (EHR) for PC risk prediction.

METHODS

Ovid MEDLINE(R), Ovid EMBASE, Ovid Cochrane Central Register of Controlled Trials, Ovid Cochrane Database of Systematic Reviews, Scopus, and Web of Science were searched for articles that utilized ML/AI techniques to predict PC, published between January 1, 2012, and February 1, 2024. Study selection and data extraction were conducted by 2 independent reviewers. Critical appraisal and data extraction were performed using the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies checklist. Risk of bias and applicability were examined using prediction model risk of bias assessment tool.

RESULTS

Thirty studies including 169,149 PC cases were identified. Logistic regression was the most frequent modeling method. Twenty studies utilized a curated set of known PC risk predictors or those identified by clinical experts. ML model discrimination performance (C-index) ranged from 0.57 to 1.0. Missing data were underreported, and most studies did not implement explainable-AI techniques or report exclusion time intervals.

DISCUSSION

AI/ML models for PC risk prediction using known risk factors perform reasonably well and may have near-term applications in identifying cohorts for targeted PC screening if validated in real-world data sets. The combined use of structured and unstructured EHR data using emerging AI models while incorporating explainable-AI techniques has the potential to identify novel PC risk factors, and this approach merits further study.

Gluten-Free Diet Adherence Tools for Individuals with Celiac Disease: A Systematic Review and Meta-Analysis of Tools Compared to Laboratory Tests

This systematic review aimed to find the tool that best predicts celiac individuals' adherence to a gluten-free diet (GFD). The Transparent Reporting of Multivariable Prediction Models for Individual Prognosis or Diagnosis (TRIPOD-SRMA) guideline was used for the construction and collection of data from eight scientific databases (PubMed, EMBASE, LILACS, Web of Science, LIVIVO, SCOPUS, Google Scholar, and Proquest) on 16 November 2023. The inclusion criteria were studies involving individuals with celiac disease (CD) who were over 18 years old and on a GFD for at least six months, using a questionnaire to predict adherence to a GFD, and comparing it with laboratory tests (serological tests, gluten immunogenic peptide-GIP, or biopsy). Review articles, book chapters, and studies without sufficient data were excluded. The Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies (CHARMS) was used for data collection from the selected primary studies, and their risk of bias and quality was assessed using the Prediction Risk of Bias Assessment Tool (PROBAST). The association between the GFD adherence determined by the tool and laboratory test was assessed using the phi contingency coefficient. The studies included in this review used four different tools to evaluate GFD adherence: BIAGI score, Coeliac Dietary Adherence Test (CDAT), self-report questions, and interviews. The comparison method most often used was biopsy (n = 19; 59.3%), followed by serology (n = 14; 43.7%) and gluten immunogenic peptides (GIPs) (n = 4; 12.5%). There were no significant differences between the interview, self-report, and BIAGI tools used to evaluate GFD adherence. These tools were better associated with GFD adherence than the CDAT. Considering their cost, application time, and prediction capacity, the self-report and BIAGI were the preferred tools for evaluating GFD adherence.

A Systematic Review of Artificial Intelligence Models for Time-to-Event Outcome Applied in Cardiovascular Disease Risk Prediction

Artificial intelligence (AI) based predictive models for early detection of cardiovascular disease (CVD) risk are increasingly being utilised. However, AI based risk prediction models that account for right-censored data have been overlooked. This systematic review (PROSPERO protocol CRD42023492655) includes 33 studies that utilised machine learning (ML) and deep learning (DL) models for survival outcome in CVD prediction. We provided details on the employed ML and DL models, eXplainable AI (XAI) techniques, and type of included variables, with a focus on social determinants of health (SDoH) and gender-stratification. Approximately half of the studies were published in 2023 with the majority from the United States. Random Survival Forest (RSF), Survival Gradient Boosting models, and Penalised Cox models were the most frequently employed ML models. DeepSurv was the most frequently employed DL model. DL models were better at predicting CVD outcomes than ML models. Permutation-based feature importance and Shapley values were the most utilised XAI methods for explaining AI models. Moreover, only one in five studies performed gender-stratification analysis and very few incorporate the wide range of SDoH factors in their prediction model. In conclusion, the evidence indicates that RSF and DeepSurv models are currently the optimal models for predicting CVD outcomes. This study also highlights the better predictive ability of DL survival models, compared to ML models. Future research should ensure the appropriate interpretation of AI models, accounting for SDoH, and gender stratification, as gender plays a significant role in CVD occurrence.

Systematic review of multivariable prognostic models for outcomes at least 30 days after hip fracture finds 18 mortality models but no nonmortality models warranting validation

OBJECTIVES

Prognostic models have the potential to aid clinical decision-making after hip fracture. This systematic review aimed to identify, critically appraise, and summarize multivariable prediction models for mortality or other long-term recovery outcomes occurring at least 30 days after hip fracture.

STUDY DESIGN AND SETTING

MEDLINE, Embase, Scopus, Web of Science, and CINAHL databases were searched up to May 2023. Studies were included that aimed to develop multivariable models to make predictions for individuals at least 30 days after hip fracture. Risk of bias (ROB) was dual-assessed using the Prediction model Risk Of Bias ASsessment Tool. Study and model details were extracted and summarized.

RESULTS

From 5571 records, 80 eligible studies were identified. They predicted mortality in n = 55 studies/81 models and nonmortality outcomes (mobility, function, residence, medical, and surgical complications) in n = 30 studies/45 models. Most (n = 46; 58%) studies were published since 2020. A quarter of studies (n = 19; 24%) reported using 'machine-learning methods', while the remainder used logistic regression (n = 54; 68%) and other statistical methods (n = 11; 14%) to build models. Overall, 15 studies (19%) presented 18 low ROB models, all predicting mortality. Common concerns were sample size, missing data handling, inadequate internal validation, and calibration assessment. Many studies with nonmortality outcomes (n = 11; 37%) had clear data complexities that were not correctly modeled.

CONCLUSION

This review has comprehensively summarized and appraised multivariable prediction models for long-term outcomes after hip fracture. Only 15 studies of 55 predicting mortality were rated as low ROB, warranting further development of their models. All studies predicting nonmortality outcomes were high or unclear ROB. Careful consideration is required for both the methods used and justification for developing further nonmortality prediction models for this clinical population.

Prognosis prediction models for post-stroke depression: a protocol for systematic review, meta-analysis, and critical appraisal

INTRODUCTION

Post-stroke depression (PSD) is a prevalent complication that has been shown to have a negative impact on rehabilitation outcomes and quality of life and poses a significant risk for suicidal intention. However, models for discriminating and predicting PSD in stroke survivors for effective secondary prevention strategies are inadequate as the pathogenesis of PSD remains unknown. Prognostic prediction models that exhibit greater rule-in capacity have the potential to mitigate the issue of underdiagnosis and undertreatment of PSD. Thus, the planned study aims to systematically review and critically evaluate published studies on prognostic prediction models for PSD.

METHODS AND ANALYSIS

A systematic literature search will be conducted in PubMed and Embase through Ovid. Two reviewers will complete study screening, data extraction, and quality assessment utilizing appropriate tools. Qualitative data on the characteristics of the included studies, methodological quality, and the appraisal of the clinical applicability of models will be summarized in the form of narrative comments and tables or figures. The predictive performance of the same model involving multiple studies will be synthesized with a random effects meta-analysis model or meta-regression, taking into account heterogeneity.

ETHICS AND DISSEMINATION

Ethical approval is considered not applicable for this systematic review. Findings will be shared through dissemination at academic conferences and/or publication in peer-reviewed academic journals.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42023388548.

A systematic review of prediction models on arteriovenous fistula: Risk scores and machine learning approaches

OBJECTIVE

Failure-to-mature and early stenosis remains the Achille's heel of hemodialysis arteriovenous fistula (AVF) creation. The maturation and patency of an AVF can be influenced by a variety of demographic, comorbidity, and anatomical factors. This study aims to review the prediction models of AVF maturation and patency with various risk scores and machine learning models.

DATA SOURCES AND REVIEW METHODS

Literature search was performed on PubMed, Scopus, and Embase to identify eligible articles. The quality of the studies was assessed using the Prediction model Risk Of Bias ASsessment (PROBAST) Tool. The performance (discrimination and calibration) of the included studies were extracted.

RESULTS

Fourteen studies (seven studies used risk score approaches; seven studies used machine learning approaches) were included in the review. Among them, 12 studies were rated as high or unclear "risk of bias." Six studies were rated as high concern or unclear for "applicability." C-statistics (Model discrimination metric) was reported in five studies using risk score approach (0.70-0.886) and three utilized machine learning methods (0.80-0.85). Model calibration was reported in three studies. Failure-to-mature risk score developed by one of the studies has been externally validated in three different patient populations, however the model discrimination degraded significantly (C-statistics: 0.519-0.53).

CONCLUSION

The performance of existing predictive models for AVF maturation/patency is underreported. They showed satisfactory performance in their own study population. However, there was high risk of bias in methodology used to build some of the models. The reviewed models also lack external validation or had reduced performance in external cohort.

Generalizable pipeline for constructing HIV risk prediction models across electronic health record systems

OBJECTIVE

The HIV epidemic remains a significant public health issue in the United States. HIV risk prediction models could be beneficial for reducing HIV transmission by helping clinicians identify patients at high risk for infection and refer them for testing. This would facilitate initiation on treatment for those unaware of their status and pre-exposure prophylaxis for those uninfected but at high risk. Existing HIV risk prediction algorithms rely on manual construction of features and are limited in their application across diverse electronic health record systems. Furthermore, the accuracy of these models in predicting HIV in females has thus far been limited.

MATERIALS AND METHODS

We devised a pipeline for automatic construction of prediction models based on automatic feature engineering to predict HIV risk and tested our pipeline on a local electronic health records system and a national claims data. We also compared the performance of general models to female-specific models.

RESULTS

Our models obtain similarly good performance on both health record datasets despite difference in represented populations and data availability (AUC = 0.87). Furthermore, our general models obtain good performance on females but are also improved by constructing female-specific models (AUC between 0.81 and 0.86 across datasets).

DISCUSSION AND CONCLUSIONS

We demonstrated that flexible construction of prediction models performs well on HIV risk prediction across diverse health records systems and perform as well in predicting HIV risk in females, making deployment of such models into existing health care systems tangible.

Using meta-analysis and CNN-NLP to review and classify the medical literature for normal tissue complication probability in head and neck cancer

PURPOSE

The study aims to enhance the efficiency and accuracy of literature reviews on normal tissue complication probability (NTCP) in head and neck cancer patients using radiation therapy. It employs meta-analysis (MA) and natural language processing (NLP).

MATERIAL AND METHODS

The study consists of two parts. First, it employs MA to assess NTCP models for xerostomia, dysphagia, and mucositis after radiation therapy, using Python 3.10.5 for statistical analysis. Second, it integrates NLP with convolutional neural networks (CNN) to optimize literature search, reducing 3256 articles to 12. CNN settings include a batch size of 50, 50-200 epoch range and a 0.001 learning rate.

RESULTS

The study's CNN-NLP model achieved a notable accuracy of 0.94 after 200 epochs with Adamax optimization. MA showed an AUC of 0.67 for early-effect xerostomia and 0.74 for late-effect, indicating moderate to high predictive accuracy but with high variability across studies. Initial CNN accuracy of 66.70% improved to 94.87% post-tuning by optimizer and hyperparameters.

CONCLUSION

The study successfully merges MA and NLP, confirming high predictive accuracy for specific model-feature combinations. It introduces a time-based metric, words per minute (WPM), for efficiency and highlights the utility of MA and NLP in clinical research.

Pre-treatment re-bleeding following aneurysmal subarachnoid hemorrhage: A systematic review of published prediction models with risk of bias and clinical applicability assessment

BACKGROUND

Pre-treatment rebleeding following aneurysmal subarachnoid hemorrhage (aSAH) increases the risk of death and a poor neurological outcome. Current guidelines recommend aneurysm treatment "as early as feasible after presentation, preferably within 24 h of onset" to mitigate this risk, a practice termed ultra-early treatment. However, ongoing debate regarding whether ultra-early treatment is independently associated with reduced re-bleeding risk, together with the recognition that re-bleeding occurs even in centres practicing ultra-early treatment due to the presence of other risk-factors has resulted in a renewed need for patient-specific re-bleed risk prediction. Here, we systematically review models which seek to provide patient specific predictions of pre-treatment rebleeding risk.

METHODS

Following registration on the International prospective register of systematic reviews (PROSPERO) CRD 42023421235; Ovid Medline (Pubmed), Embase and Googlescholar were searched for English language studies between 1st May 2002 and 1st June 2023 describing pre-treatment rebleed prediction models following aSAH in adults ≥18 years. Of 763 unique records, 17 full texts were scrutinised with 5 publications describing 4 models reviewed. We used the semi-automated template of Fernandez-Felix et al. incorporating the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist and the Prediction model Risk Of Bias ASsessment Tool (PROBAST) for data extraction, risk of bias and clinical applicability assessment. To further standardize risk of bias and clinical applicability assessment, we also used the published explanatory notes for the PROBAST tool and compared the aneurysm treatment practices each prediction model's formulation cohort experienced to a prespecified benchmark representative of contemporary aneurysm treatment practices as outlined in recent evidence-based guidelines and published practice pattern reports from four developed countries.

RESULTS

Reported model discriminative performance varied between 0.77 and 0.939, however, no single model demonstrated a consistently low risk of bias and low concern for clinical applicability in all domains. Only the score of Darkwah Oppong et al. was formulated using a patient cohort in which the majority of patients were managed in accordance with contemporary, evidence-based aneurysm treatment practices defined by ultra-early and predominantly endovascular treatment. However, this model did not undergo calibration or clinical utility analysis and when applied to an external cohort, its discriminative performance was substantially lower that reported at formulation.

CONCLUSIONS

No existing prediction model can be recommended for clinical use in centers practicing contemporary, evidence-based aneurysm treatment. There is a pressing need for improved prediction models to estimate and minimize pre-treatment re-bleeding risk.

Artificial Intelligence in Predicting Microsatellite Instability and KRAS, BRAF Mutations from Whole-Slide Images in Colorectal Cancer: A Systematic Review

Mismatch repair deficiency (d-MMR)/microsatellite instability (MSI), KRAS, and BRAF mutational status are crucial for treating advanced colorectal cancer patients. Traditional methods like immunohistochemistry or polymerase chain reaction (PCR) can be challenged by artificial intelligence (AI) based on whole slide images (WSI) to predict tumor status. In this systematic review, we evaluated the role of AI in predicting MSI status, KRAS, and BRAF mutations in colorectal cancer. Studies published in PubMed up to June 2023 were included (n = 17), and we reported the risk of bias and the performance for each study. Some studies were impacted by the reduced number of slides included in the data set and the lack of external validation cohorts. Deep learning models for the d-MMR/MSI status showed a good performance in training cohorts (mean AUC = 0.89, [0.74-0.97]) but slightly less than expected in the validation cohort when available (mean AUC = 0.82, [0.63-0.98]). Contrary to the MSI status, the prediction of KRAS and BRAF mutations was less explored with a less robust methodology. The performance was lower, with a maximum of 0.77 in the training cohort, 0.58 in the validation cohort for KRAS, and 0.82 AUC in the training cohort for BRAF.

Technical/Algorithm, Stakeholder, and Society (TASS) barriers to the application of artificial intelligence in medicine: A systematic review

INTRODUCTION

The use of artificial intelligence (AI), particularly machine learning and predictive analytics, has shown great promise in health care. Despite its strong potential, there has been limited use in health care settings. In this systematic review, we aim to determine the main barriers to successful implementation of AI in healthcare and discuss potential ways to overcome these challenges.

METHODS

We conducted a literature search in PubMed (1/1/2001-1/1/2023). The search was restricted to publications in the English language, and human study subjects. We excluded articles that did not discuss AI, machine learning, predictive analytics, and barriers to the use of these techniques in health care. Using grounded theory methodology, we abstracted concepts to identify major barriers to AI use in medicine.

RESULTS

We identified a total of 2,382 articles. After reviewing the 306 included papers, we developed 19 major themes, which we categorized into three levels: the Technical/Algorithm, Stakeholder, and Social levels (TASS). These themes included: Lack of Explainability, Need for Validation Protocols, Need for Standards for Interoperability, Need for Reporting Guidelines, Need for Standardization of Performance Metrics, Lack of Plan for Updating Algorithm, Job Loss, Skills Loss, Workflow Challenges, Loss of Patient Autonomy and Consent, Disturbing the Patient-Clinician Relationship, Lack of Trust in AI, Logistical Challenges, Lack of strategic plan, Lack of Cost-effectiveness Analysis and Proof of Efficacy, Privacy, Liability, Bias and Social Justice, and Education.

CONCLUSION

We identified 19 major barriers to the use of AI in healthcare and categorized them into three levels: the Technical/Algorithm, Stakeholder, and Social levels (TASS). Future studies should expand on barriers in pediatric care and focus on developing clearly defined protocols to overcome these barriers.

Systematic reviewers used various approaches to data extraction and expressed several research needs: a survey

OBJECTIVE

Data extraction is a prerequisite for analyzing, summarizing, and interpreting evidence in systematic reviews. Yet guidance is limited, and little is known about current approaches. We surveyed systematic reviewers on their current approaches to data extraction, opinions on methods, and research needs.

STUDY DESIGN AND SETTING

We developed a 29-question online survey and distributed it through relevant organizations, social media, and personal networks in 2022. Closed questions were evaluated using descriptive statistics, and open questions were analyzed using content analysis.

RESULTS

162 reviewers participated. Use of adapted (65%) or newly developed extraction forms (62%) was common. Generic forms were rarely used (14%). Spreadsheet software was the most popular extraction tool (83%). Piloting was reported by 74% of respondents and included a variety of approaches. Independent and duplicate extraction was considered the most appropriate approach to data collection (64%). About half of respondents agreed that blank forms and/or raw data should be published. Suggested research gaps were the effects of different methods on error rates (60%) and the use of data extraction support tools (46%).

CONCLUSION

Systematic reviewers used varying approaches to pilot data extraction. Methods to reduce errors and use of support tools such as (semi-)automation tools are top research gaps.