1
|
Zhong K, An X, Kong Y, Chen Z. Predictive model for the risk of hemorrhagic transformation after rt-PA intravenous thrombolysis in patients with acute ischemic stroke: A systematic review and meta-analysis. Clin Neurol Neurosurg 2024; 239:108225. [PMID: 38479035 DOI: 10.1016/j.clineuro.2024.108225] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2023] [Revised: 01/15/2024] [Accepted: 03/02/2024] [Indexed: 04/02/2024]
Abstract
OBJECTIVE To systematically review the risk prediction model of Hemorrhages Transformation (HT) after intravenous thrombolysis in patients with Acute Ischemic Stroke (AIS). METHODS Web of Science, The Cochrane Library, PubMed, Embase, CINAHL, CNKI, CBM, WanFang, and VIP were searched from inception to February 25, 2023 for literature related to the risk prediction model for HT after thrombolysis in AIS. RESULTS A total of 17 included studies contained 26 prediction models, and the AUC of all models at the time of modeling ranged from 0.662 to 0.9854, 16 models had AUC>0.8, indicating that the models had good predictive performance. However, most of the included studies were at risk of bias. the results of the Meta-analysis showed that atrial fibrillation (OR=2.72, 95% CI:1.98-3.73), NIHSS score (OR=1.09, 95% CI:1.07-1.11), glucose (OR=1.12, 95% CI:1.06-1.18), moderate to severe leukoaraiosis (OR=3.47, 95% CI:1.61-7.52), hyperdense middle cerebral artery sign (OR=2.35, 95% CI:1.10-4.98), large cerebral infarction (OR=7.57, 95% CI:2.09-27.43), and early signs of infarction (OR=4.80, 95% CI:1.74-13.25) were effective predictors of HT after intravenous thrombolysis in patients with AIS. CONCLUSIONS The performance of the models for HT after thrombolysis in patients with AIS in the Chinese population is good, but there is some risk of bias. Future post-intravenous HT conversion prediction models for AIS patients in the Chinese population should focus on predictors such as atrial fibrillation, NIHSS score, glucose, moderate to severe leukoaraiosis, hyperdense middle cerebral artery sign, massive cerebral infarction, and early signs of infarction.
Collapse
Affiliation(s)
- Kelong Zhong
- Chengdu University of Traditional Chinese Medicine, China
| | - Xuemei An
- Hospital of Chengdu University of Traditional Chinese Medicine, China.
| | - Yun Kong
- Chengdu University of Traditional Chinese Medicine, China
| | - Zhu Chen
- Sichuan Provincial Maternity and Child Health Care Hospital, China
| |
Collapse
|
2
|
Pacchiano F, Tortora M, Criscuolo S, Jaber K, Acierno P, De Simone M, Tortora F, Briganti F, Caranci F. Artificial intelligence applied in acute ischemic stroke: from child to elderly. LA RADIOLOGIA MEDICA 2024; 129:83-92. [PMID: 37878222 PMCID: PMC10808481 DOI: 10.1007/s11547-023-01735-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/17/2023] [Accepted: 09/28/2023] [Indexed: 10/26/2023]
Abstract
This review will summarize artificial intelligence developments in acute ischemic stroke in recent years and forecasts for the future. Stroke is a major healthcare concern due to its effects on the patient's quality of life and its dependence on the timing of the identification as well as the treatment. In recent years, attention increased on the use of artificial intelligence (AI) systems to help categorize, prognosis, and to channel these patients toward the right therapeutic procedure. Machine learning (ML) and in particular deep learning (DL) systems using convoluted neural networks (CNN) are becoming increasingly popular. Various studies over the years evaluated the use of these methods of analysis and prediction in the assessment of stroke patients, and at the same time, several applications and software have been developed to support the neuroradiologists and the stroke team to improve patient outcomes.
Collapse
Affiliation(s)
- Francesco Pacchiano
- Department of Precision Medicine, University of Campania "L. Vanvitelli", Caserta, Italy
| | - Mario Tortora
- Department of Advanced Biomedical Sciences, University "Federico II", Via Pansini, 5, 80131, Naples, Italy.
| | - Sabrina Criscuolo
- Pediatric University Department, Bambino Gesù Children Hospital, Rome, Italy
| | - Katya Jaber
- Department of Elektrotechnik und Informatik, Hochschule Bremen, Bremen, Germany
| | | | - Marta De Simone
- UOC Neuroradiology, AORN San Giuseppe Moscati, Avellino, Italy
| | - Fabio Tortora
- Department of Advanced Biomedical Sciences, University "Federico II", Via Pansini, 5, 80131, Naples, Italy
| | - Francesco Briganti
- Department of Advanced Biomedical Sciences, University "Federico II", Via Pansini, 5, 80131, Naples, Italy
| | - Ferdinando Caranci
- Department of Precision Medicine, University of Campania "L. Vanvitelli", Caserta, Italy
| |
Collapse
|
3
|
Ru X, Zhao S, Chen W, Wu J, Yu R, Wang D, Dong M, Wu Q, Peng D, Song Y. A weakly supervised deep learning model integrating noncontrasted computed tomography images and clinical factors facilitates haemorrhagic transformation prediction after intravenous thrombolysis in acute ischaemic stroke patients. Biomed Eng Online 2023; 22:129. [PMID: 38115029 PMCID: PMC10731772 DOI: 10.1186/s12938-023-01193-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2023] [Accepted: 12/09/2023] [Indexed: 12/21/2023] Open
Abstract
BACKGROUND Haemorrhage transformation (HT) is a serious complication of intravenous thrombolysis (IVT) in acute ischaemic stroke (AIS). Accurate and timely prediction of the risk of HT before IVT may change the treatment decision and improve clinical prognosis. We aimed to develop a deep learning method for predicting HT after IVT for AIS using noncontrast computed tomography (NCCT) images. METHODS We retrospectively collected data from 828 AIS patients undergoing recombinant tissue plasminogen activator (rt-PA) treatment within a 4.5-h time window (n = 665) or of undergoing urokinase treatment within a 6-h time window (n = 163) and divided them into the HT group (n = 69) and non-HT group (n = 759). HT was defined based on the criteria of the European Cooperative Acute Stroke Study-II trial. To address the problems of indiscernible features and imbalanced data, a weakly supervised deep learning (WSDL) model for HT prediction was constructed based on multiple instance learning and active learning using admission NCCT images and clinical information in addition to conventional deep learning models. Threefold cross-validation and transfer learning were performed to confirm the robustness of the network. Of note, the predictive value of the commonly used scales in clinics associated with NCCT images (i.e., the HAT and SEDAN score) was also analysed and compared to measure the feasibility of our proposed DL algorithms. RESULTS Compared to the conventional DL and ML models, the WSDL model had the highest AUC of 0.799 (95% CI 0.712-0.883). Significant differences were observed between the WSDL model and five ML models (P < 0.05). The prediction performance of the WSDL model outperforms the HAT and SEDAN scores at the optimal operating point (threshold = 1.5). Further subgroup analysis showed that the WSDL model performed better for symptomatic intracranial haemorrhage (AUC = 0.833, F1 score = 0.909). CONCLUSIONS Our WSDL model based on NCCT images had relatively good performance for predicting HT in AIS and may be suitable for assisting in clinical treatment decision-making.
Collapse
Affiliation(s)
- Xiaoshuang Ru
- Department of Radiology, Central Hospital of Dalian University of Technology, No. 826 Xinan Rd, Shahekou District, Dalian, 116033, Liaoning Province, China
| | - Shilong Zhao
- Department of Radiology, Affliated ZhongShan Hospital of Dalian University, No. 6 Jiefang Rd, Zhongshan District, Dalian, 116001, Liaoning Province, China
| | - Weidao Chen
- InferVision Medical Technology Company Ltd, 25F, Building E, Yuanyang International Center, Chaoyang District, Beijing, 100025, China
| | - Jiangfen Wu
- InferVision Medical Technology Company Ltd, 25F, Building E, Yuanyang International Center, Chaoyang District, Beijing, 100025, China
| | - Ruize Yu
- InferVision Medical Technology Company Ltd, 25F, Building E, Yuanyang International Center, Chaoyang District, Beijing, 100025, China
| | - Dawei Wang
- InferVision Medical Technology Company Ltd, 25F, Building E, Yuanyang International Center, Chaoyang District, Beijing, 100025, China
| | - Mengxing Dong
- InferVision Medical Technology Company Ltd, 25F, Building E, Yuanyang International Center, Chaoyang District, Beijing, 100025, China
| | - Qiong Wu
- Department of Neurology, Central Hospital of Dalian University of Technology, No. 826 Xinan Rd, Shahekou District, Dalian, 116033, Liaoning Province, China
| | - Daoyong Peng
- Department of Neurology, Central Hospital of Dalian University of Technology, No. 826 Xinan Rd, Shahekou District, Dalian, 116033, Liaoning Province, China
| | - Yang Song
- Department of Radiology, Central Hospital of Dalian University of Technology, No. 826 Xinan Rd, Shahekou District, Dalian, 116033, Liaoning Province, China.
| |
Collapse
|
4
|
Wen R, Wang M, Bian W, Zhu H, Xiao Y, He Q, Wang Y, Liu X, Shi Y, Hong Z, Xu B. Machine learning-based prediction of symptomatic intracerebral hemorrhage after intravenous thrombolysis for stroke: a large multicenter study. Front Neurol 2023; 14:1247492. [PMID: 37928151 PMCID: PMC10624225 DOI: 10.3389/fneur.2023.1247492] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2023] [Accepted: 09/28/2023] [Indexed: 11/07/2023] Open
Abstract
Background This study aimed to compare the performance of different machine learning models in predicting symptomatic intracranial hemorrhage (sICH) after thrombolysis treatment for ischemic stroke. Methods This multicenter study utilized the Shenyang Stroke Emergency Map database, comprising 8,924 acute ischemic stroke patients from 29 comprehensive hospitals who underwent thrombolysis between January 2019 and December 2021. An independent testing cohort was further established, including 1,921 patients from the First People's Hospital of Shenyang. The structured dataset encompassed 15 variables, including clinical and therapeutic metrics. The primary outcome was the sICH occurrence post-thrombolysis. Models were developed using an 80/20 split for training and internal validation. Performance was assessed using machine learning classifiers, including logistic regression with lasso regularization, support vector machine (SVM), random forest, gradient-boosted decision tree (GBDT), and multilayer perceptron (MLP). The model boasting the highest area under the curve (AUC) was specifically employed to highlight feature importance. Results Baseline characteristics were compared between the training cohort (n = 6,369) and the external validation cohort (n = 1,921), with the sICH incidence being slightly higher in the training cohort (1.6%) compared to the validation cohort (1.1%). Among the evaluated models, the logistic regression with lasso regularization achieved the highest AUC of 0.87 (95% confidence interval [CI]: 0.79-0.95; p < 0.001), followed by the MLP model with an AUC of 0.766 (95% CI: 0.637-0.894; p = 0.04). The reference model and SVM showed AUCs of 0.575 and 0.582, respectively, while the random forest and GBDT models performed less optimally with AUCs of 0.536 and 0.436, respectively. Decision curve analysis revealed net benefits primarily for the SVM and MLP models. Feature importance from the logistic regression model emphasized anticoagulation therapy as the most significant negative predictor (coefficient: -2.0833) and recombinant tissue plasminogen activator as the principal positive predictor (coefficient: 0.5082). Conclusion After a comprehensive evaluation, the MLP model is recommended due to its superior ability to predict the risk of symptomatic hemorrhage post-thrombolysis in ischemic stroke patients. Based on decision curve analysis, the MLP-based model was chosen and demonstrated enhanced discriminative ability compared to the reference. This model serves as a valuable tool for clinicians, aiding in treatment planning and ensuring more precise forecasting of patient outcomes.
Collapse
Affiliation(s)
- Rui Wen
- Shenyang Tenth People’s Hospital, Shenyang, China
| | - Miaoran Wang
- Affiliated Central Hospital of Shenyang Medical College, Shenyang Medical College, Shenyang, China
| | - Wei Bian
- Shenyang First People’s Hospital, Shenyang Medical College, Shenyang, China
| | - Haoyue Zhu
- Shenyang First People’s Hospital, Shenyang Medical College, Shenyang, China
| | - Ying Xiao
- Shenyang First People’s Hospital, Shenyang Medical College, Shenyang, China
| | - Qian He
- Shenyang Tenth People’s Hospital, Shenyang, China
| | - Yu Wang
- Shenyang Tenth People’s Hospital, Shenyang, China
| | - Xiaoqing Liu
- Shenyang Tenth People’s Hospital, Shenyang, China
| | - Yangdi Shi
- Shenyang Tenth People’s Hospital, Shenyang, China
| | - Zhe Hong
- Shenyang First People’s Hospital, Shenyang Medical College, Shenyang, China
| | - Bing Xu
- Shenyang Tenth People’s Hospital, Shenyang, China
| |
Collapse
|
5
|
Zhang K, Jiang Y, Zeng H, Zhu H. Application and risk prediction of thrombolytic therapy in cardio-cerebrovascular diseases: a review. Thromb J 2023; 21:90. [PMID: 37667349 PMCID: PMC10476453 DOI: 10.1186/s12959-023-00532-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Accepted: 08/18/2023] [Indexed: 09/06/2023] Open
Abstract
Cardiocerebrovascular diseases (CVDs) are the leading cause of death worldwide, consuming huge healthcare budget. For CVD patients, the prompt assessment and appropriate administration is the crux to save life and improve prognosis. Thrombolytic therapy, as a non-invasive approach to achieve recanalization, is the basic component of CVD treatment. Still, there are risks that limits its application. The objective of this review is to give an introduction on the utilization of thrombolytic therapy in cardiocerebrovascular blockage diseases, including coronary heart disease and ischemic stroke, and to review the development in risk assessment of thrombolytic therapy, comparing the performance of traditional scales and novel artificial intelligence-based risk assessment models.
Collapse
Affiliation(s)
- Kexin Zhang
- Division of Cardiology, Department of Internal Medicine, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, 430030, Hubei, China
| | - Yao Jiang
- Division of Cardiology, Department of Internal Medicine, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, 430030, Hubei, China
| | - Hesong Zeng
- Division of Cardiology, Department of Internal Medicine, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, 430030, Hubei, China
| | - Hongling Zhu
- Division of Cardiology, Department of Internal Medicine, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, 430030, Hubei, China.
| |
Collapse
|
6
|
Akay EMZ, Hilbert A, Carlisle BG, Madai VI, Mutke MA, Frey D. Artificial Intelligence for Clinical Decision Support in Acute Ischemic Stroke: A Systematic Review. Stroke 2023; 54:1505-1516. [PMID: 37216446 DOI: 10.1161/strokeaha.122.041442] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Accepted: 02/21/2023] [Indexed: 05/24/2023]
Abstract
BACKGROUND Established randomized trial-based parameters for acute ischemic stroke group patients into generic treatment groups, leading to attempts using various artificial intelligence (AI) methods to directly correlate patient characteristics to outcomes and thereby provide decision support to stroke clinicians. We review AI-based clinical decision support systems in the development stage, specifically regarding methodological robustness and constraints for clinical implementation. METHODS Our systematic review included full-text English language publications proposing a clinical decision support system using AI techniques for direct decision support in acute ischemic stroke cases in adult patients. We (1) describe data and outcomes used in those systems, (2) estimate the systems' benefits compared with traditional stroke diagnosis and treatment, and (3) reported concordance with reporting standards for AI in healthcare. RESULTS One hundred twenty-one studies met our inclusion criteria. Sixty-five were included for full extraction. In our sample, utilized data sources, methods, and reporting practices were highly heterogeneous. CONCLUSIONS Our results suggest significant validity threats, dissonance in reporting practices, and challenges to clinical translation. We outline practical recommendations for the successful implementation of AI research in acute ischemic stroke treatment and diagnosis.
Collapse
Affiliation(s)
- Ela Marie Z Akay
- Charité Lab for Artificial Intelligence in Medicine (CLAIM) (E.M.Z.A., A.H., D.F.), Charité Universitätsmedizin Berlin, Germany
| | - Adam Hilbert
- Charité Lab for Artificial Intelligence in Medicine (CLAIM) (E.M.Z.A., A.H., D.F.), Charité Universitätsmedizin Berlin, Germany
| | - Benjamin G Carlisle
- QUEST Center for Responsible Research, Berlin Institute of Health (BIH) (B.G.C., V.I.M.), Charité Universitätsmedizin Berlin, Germany
| | - Vince I Madai
- QUEST Center for Responsible Research, Berlin Institute of Health (BIH) (B.G.C., V.I.M.), Charité Universitätsmedizin Berlin, Germany
- Faculty of Computing, Engineering and the Built Environment, School of Computing and Digital Technology, Birmingham City University, United Kingdom (V.I.M.)
| | - Matthias A Mutke
- Department of Neuroradiology, Heidelberg University Hospital, Germany (M.A.M.)
| | - Dietmar Frey
- Charité Lab for Artificial Intelligence in Medicine (CLAIM) (E.M.Z.A., A.H., D.F.), Charité Universitätsmedizin Berlin, Germany
| |
Collapse
|
7
|
Ren H, Song H, Wang J, Xiong H, Long B, Gong M, Liu J, He Z, Liu L, Jiang X, Li L, Li H, Cui S, Li Y. A clinical-radiomics model based on noncontrast computed tomography to predict hemorrhagic transformation after stroke by machine learning: a multicenter study. Insights Imaging 2023; 14:52. [PMID: 36977913 PMCID: PMC10050271 DOI: 10.1186/s13244-023-01399-5] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2022] [Accepted: 03/08/2023] [Indexed: 03/30/2023] Open
Abstract
OBJECTIVE To build a clinical-radiomics model based on noncontrast computed tomography images to identify the risk of hemorrhagic transformation (HT) in patients with acute ischemic stroke (AIS) following intravenous thrombolysis (IVT). MATERIALS AND METHODS A total of 517 consecutive patients with AIS were screened for inclusion. Datasets from six hospitals were randomly divided into a training cohort and an internal cohort with an 8:2 ratio. The dataset of the seventh hospital was used for an independent external verification. The best dimensionality reduction method to choose features and the best machine learning (ML) algorithm to develop a model were selected. Then, the clinical, radiomics and clinical-radiomics models were developed. Finally, the performance of the models was measured using the area under the receiver operating characteristic curve (AUC). RESULTS Of 517 from seven hospitals, 249 (48%) had HT. The best method for choosing features was recursive feature elimination, and the best ML algorithm to build models was extreme gradient boosting. In distinguishing patients with HT, the AUC of the clinical model was 0.898 (95% CI 0.873-0.921) in the internal validation cohort, and 0.911 (95% CI 0.891-0.928) in the external validation cohort; the AUC of radiomics model was 0.922 (95% CI 0.896-0.941) and 0.883 (95% CI 0.851-0.902), while the AUC of clinical-radiomics model was 0.950 (95% CI 0.925-0.967) and 0.942 (95% CI 0.927-0.958) respectively. CONCLUSION The proposed clinical-radiomics model is a dependable approach that could provide risk assessment of HT for patients who receive IVT after stroke.
Collapse
Affiliation(s)
- Huanhuan Ren
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, No. 1 Youyi Road, Yuzhong District, Chongqing, 400016, China
- Department of Radiology, Chongqing General Hospital, Chongqing, China
| | - Haojie Song
- College of Computer and Information Science, Chongqing Normal University, No. 37, Middle University Town Road, Shapingba District, Chongqing, 400016, China
| | - Jingjie Wang
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, No. 1 Youyi Road, Yuzhong District, Chongqing, 400016, China
| | - Hua Xiong
- Department of Radiology, Chongqing General Hospital, Chongqing, China
| | - Bangyuan Long
- Department of Radiology, Chongqing General Hospital, Chongqing, China
| | - Meilin Gong
- Department of Radiology, Chongqing General Hospital, Chongqing, China
| | - Jiayang Liu
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, No. 1 Youyi Road, Yuzhong District, Chongqing, 400016, China
| | - Zhanping He
- Department of Radiology, Haikou Affiliated Hospital of Central South University Xiangya School of Medicine, Haikou, China
| | - Li Liu
- Department of Radiology, People's Hospital of Yubei District of Chongqing City, Chongqing, China
| | - Xili Jiang
- Department of Radiology, The Second People's Hospital of Hunan Province/Brain Hospital of Hunan Province, Changsha, China
| | - Lifeng Li
- Department of Radiology, Changsha Central Hospital (The Affiliated Changsha Central Hospital, Hengyang Medical School, University of South China), Changsha, China
| | - Hanjian Li
- Department of Radiology, The First Affiliated Hospital of Hainan Medical University, Haikou, China
| | - Shaoguo Cui
- College of Computer and Information Science, Chongqing Normal University, No. 37, Middle University Town Road, Shapingba District, Chongqing, 400016, China.
| | - Yongmei Li
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, No. 1 Youyi Road, Yuzhong District, Chongqing, 400016, China.
| |
Collapse
|
8
|
van der Ende NA, Kremers FC, van der Steen W, Venema E, Kappelhof M, Majoie CB, Postma AA, Boiten J, van den Wijngaard IR, van der Lugt A, Dippel DW, Roozenbeek B. Symptomatic Intracranial Hemorrhage After Endovascular Stroke Treatment: External Validation of Prediction Models. Stroke 2023; 54:476-487. [PMID: 36689584 PMCID: PMC9855739 DOI: 10.1161/strokeaha.122.040065] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2022] [Accepted: 12/09/2022] [Indexed: 01/24/2023]
Abstract
BACKGROUND Symptomatic intracranial hemorrhage (sICH) is a severe complication of reperfusion therapy for ischemic stroke. Multiple models have been developed to predict sICH or intracranial hemorrhage (ICH) after reperfusion therapy. We provide an overview of published models and validate their ability to predict sICH in patients treated with endovascular treatment in daily clinical practice. METHODS We conducted a systematic search to identify models either developed or validated to predict sICH or ICH after reperfusion therapy (intravenous thrombolysis and/or endovascular treatment) for ischemic stroke. Models were externally validated in the MR CLEAN Registry (n=3180; Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands). The primary outcome was sICH according to the Heidelberg Bleeding Classification. Model performance was evaluated with discrimination (c-statistic, ideally 1; a c-statistic below 0.7 is considered poor in discrimination) and calibration (slope, ideally 1, and intercept, ideally 0). RESULTS We included 39 studies describing 40 models. The most frequently used predictors were baseline National Institutes of Health Stroke Scale (NIHSS; n=35), age (n=22), and glucose level (n=22). In the MR CLEAN Registry, sICH occurred in 188/3180 (5.9%) patients. Discrimination ranged from 0.51 (SPAN-100 [Stroke Prognostication Using Age and National Institutes of Health Stroke Scale]) to 0.61 (SITS-SICH [Safe Implementation of Treatments in Stroke Symptomatic Intracerebral Hemorrhage] and STARTING-SICH [STARTING Symptomatic Intracerebral Hemorrhage]). Best calibrated models were IST-3 (intercept, -0.15 [95% CI, -0.01 to -0.31]; slope, 0.80 [95% CI, 0.50-1.09]), SITS-SICH (intercept, 0.15 [95% CI, -0.01 to 0.30]; slope, 0.62 [95% CI, 0.38-0.87]), and STARTING-SICH (intercept, -0.03 [95% CI, -0.19 to 0.12]; slope, 0.56 [95% CI, 0.35-0.76]). CONCLUSIONS The investigated models to predict sICH or ICH discriminate poorly between patients with a low and high risk of sICH after endovascular treatment in daily clinical practice and are, therefore, not clinically useful for this patient population.
Collapse
Affiliation(s)
- Nadinda A.M. van der Ende
- Departments of Neurology (N.A.M.v.d.E, F.C.C.K., W.v.d.S, E.V., D.W.J.D., B.R.), Erasmus MC University Medical Center, the Netherlands
- Radiology and Nuclear Medicine (N.A.M.v.d.Ee, W.v.d.S., B.R.), Erasmus MC University Medical Center, the Netherlands
| | - Femke C.C. Kremers
- Departments of Neurology (N.A.M.v.d.E, F.C.C.K., W.v.d.S, E.V., D.W.J.D., B.R.), Erasmus MC University Medical Center, the Netherlands
| | - Wouter van der Steen
- Departments of Neurology (N.A.M.v.d.E, F.C.C.K., W.v.d.S, E.V., D.W.J.D., B.R.), Erasmus MC University Medical Center, the Netherlands
- Radiology and Nuclear Medicine (N.A.M.v.d.Ee, W.v.d.S., B.R.), Erasmus MC University Medical Center, the Netherlands
| | - Esmee Venema
- Departments of Neurology (N.A.M.v.d.E, F.C.C.K., W.v.d.S, E.V., D.W.J.D., B.R.), Erasmus MC University Medical Center, the Netherlands
- Emergency Medicine (E.V.), Erasmus MC University Medical Center, the Netherlands
| | - Manon Kappelhof
- Department of Radiology and Nuclear Medicine (M.K.), Amsterdam UMC, University of Amsterdam, the Netherlands
| | - Charles B.L.M. Majoie
- Departments of Neurology (N.A.M.v.d.E, F.C.C.K., W.v.d.S, E.V., D.W.J.D., B.R.), Erasmus MC University Medical Center, the Netherlands
- Radiology and Nuclear Medicine (N.A.M.v.d.Ee, W.v.d.S., B.R.), Erasmus MC University Medical Center, the Netherlands
- Emergency Medicine (E.V.), Erasmus MC University Medical Center, the Netherlands
- Department of Radiology and Nuclear Medicine (M.K.), Amsterdam UMC, University of Amsterdam, the Netherlands
- Department of Radiology and Nuclear Medicine, School for Mental Health and Sciences, Maastricht University Medical Center, the Netherlands (A.A.P.)
- Departments of Neurology (J.B., I.R.v.d.W.), Haaglanden Medical Center, the Netherlands
- Radiology and Nuclear Medicine (I.R.v.d.W.), Haaglanden Medical Center, the Netherlands
| | - Alida A. Postma
- Department of Radiology and Nuclear Medicine, School for Mental Health and Sciences, Maastricht University Medical Center, the Netherlands (A.A.P.)
| | - Jelis Boiten
- Departments of Neurology (J.B., I.R.v.d.W.), Haaglanden Medical Center, the Netherlands
| | - Ido R. van den Wijngaard
- Departments of Neurology (J.B., I.R.v.d.W.), Haaglanden Medical Center, the Netherlands
- Radiology and Nuclear Medicine (I.R.v.d.W.), Haaglanden Medical Center, the Netherlands
| | - Aad van der Lugt
- Departments of Neurology (N.A.M.v.d.E, F.C.C.K., W.v.d.S, E.V., D.W.J.D., B.R.), Erasmus MC University Medical Center, the Netherlands
- Radiology and Nuclear Medicine (N.A.M.v.d.Ee, W.v.d.S., B.R.), Erasmus MC University Medical Center, the Netherlands
- Emergency Medicine (E.V.), Erasmus MC University Medical Center, the Netherlands
- Department of Radiology and Nuclear Medicine (M.K.), Amsterdam UMC, University of Amsterdam, the Netherlands
- Department of Radiology and Nuclear Medicine, School for Mental Health and Sciences, Maastricht University Medical Center, the Netherlands (A.A.P.)
- Departments of Neurology (J.B., I.R.v.d.W.), Haaglanden Medical Center, the Netherlands
- Radiology and Nuclear Medicine (I.R.v.d.W.), Haaglanden Medical Center, the Netherlands
| | - Diederik W.J. Dippel
- Departments of Neurology (N.A.M.v.d.E, F.C.C.K., W.v.d.S, E.V., D.W.J.D., B.R.), Erasmus MC University Medical Center, the Netherlands
| | - Bob Roozenbeek
- Departments of Neurology (N.A.M.v.d.E, F.C.C.K., W.v.d.S, E.V., D.W.J.D., B.R.), Erasmus MC University Medical Center, the Netherlands
- Radiology and Nuclear Medicine (N.A.M.v.d.Ee, W.v.d.S., B.R.), Erasmus MC University Medical Center, the Netherlands
| |
Collapse
|
9
|
Shao H, Chan WCL, Du H, Chen XF, Ma Q, Shao Z. A new machine learning algorithm with high interpretability for improving the safety and efficiency of thrombolysis for stroke patients: A hospital-based pilot study. Digit Health 2023; 9:20552076221149528. [PMID: 36636727 PMCID: PMC9829886 DOI: 10.1177/20552076221149528] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/04/2023] Open
Abstract
Background Thrombolysis is the first-line treatment for patients with acute ischemic stroke. Previous studies leveraged machine learning to assist neurologists in selecting patients who could benefit the most from thrombolysis. However, when designing the algorithm, most of the previous algorithms traded interpretability for predictive power, making the algorithms hard to be trusted by neurologists and be used in real clinical practice. Methods Our proposed algorithm is an advanced version of classical k-nearest neighbors classification algorithm (KNN). We achieved high interpretability by changing the isotropy in feature space of classical KNN. We leveraged a cohort of 189 patients to prove that our algorithm maintains the interpretability of previous models while in the meantime improving the predictive power when compared with the existing algorithms. The predictive powers of models were assessed by area under the receiver operating characteristic curve (AUC). Results In terms of interpretability, only onset time, diabetes, and baseline National Institutes of Health Stroke Scale (NIHSS) were statistically significant and their contributions to the final prediction were forced to be proportional to their feature importance values by the rescaling formula we defined. In terms of predictive power, our advanced KNN (AUC 0.88) outperformed the classical KNN (AUC 0.75, p = 0.0192 ). Conclusions Our preliminary results show that the advanced KNN achieved high AUC and identified consistent significant clinical features as previous clinical trials/observational studies did. This model shows the potential to assist in thrombolysis patient selection for improving the successful rate of thrombolysis.
Collapse
Affiliation(s)
- Huiling Shao
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Kowloon, Hong Kong,Huiling Shao, Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Room Y934, 9/F, Lee Shau Kee Building, Hung Hom, Kowloon, 999077, Hong Kong.
| | - Wing Chi Lawrence Chan
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Kowloon, Hong Kong
| | - Heng Du
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Kowloon, Hong Kong
| | - Xiangyan Fiona Chen
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Kowloon, Hong Kong
| | - Qilin Ma
- Department of Neurology, The First Affiliated Hospital of Xiamen University, Xiamen, China
| | - Zhiyu Shao
- Department of Neurology, The First Affiliated Hospital of Xiamen University, Xiamen, China
| |
Collapse
|
10
|
Shao H, Chen X, Ma Q, Shao Z, Du H, Chan LWC. The feasibility and accuracy of machine learning in improving safety and efficiency of thrombolysis for patients with stroke: Literature review and proposed improvements. Front Neurol 2022; 13:934929. [PMID: 36341121 PMCID: PMC9630915 DOI: 10.3389/fneur.2022.934929] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2022] [Accepted: 09/28/2022] [Indexed: 11/30/2022] Open
Abstract
In the treatment of ischemic stroke, timely and efficient recanalization of occluded brain arteries can successfully salvage the ischemic brain. Thrombolysis is the first-line treatment for ischemic stroke. Machine learning models have the potential to select patients who could benefit the most from thrombolysis. In this study, we identified 29 related previous machine learning models, reviewed the models on the accuracy and feasibility, and proposed corresponding improvements. Regarding accuracy, lack of long-term outcome, treatment option consideration, and advanced radiological features were found in many previous studies in terms of model conceptualization. Regarding interpretability, most of the previous models chose restrictive models for high interpretability and did not mention processing time consideration. In the future, model conceptualization could be improved based on comprehensive neurological domain knowledge and feasibility needs to be achieved by elaborate computer science algorithms to increase the interpretability of flexible algorithms and shorten the processing time of the pipeline interpreting medical images.
Collapse
Affiliation(s)
- Huiling Shao
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong, Hong Kong SAR, China
| | - Xiangyan Chen
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong, Hong Kong SAR, China
| | - Qilin Ma
- Department of Neurology, The First Affiliated Hospital of Xiamen University, Xiamen, China
| | - Zhiyu Shao
- Department of Neurology, The First Affiliated Hospital of Xiamen University, Xiamen, China
| | - Heng Du
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong, Hong Kong SAR, China
| | - Lawrence Wing Chi Chan
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong, Hong Kong SAR, China
- *Correspondence: Lawrence Wing Chi Chan
| |
Collapse
|
11
|
Liu J, Chen X, Guo X, Xu R, Wang Y, Liu M. Machine learning prediction of symptomatic intracerebral hemorrhage after stroke thrombolysis: a cross-cultural validation in Caucasian and Han Chinese cohort. Ther Adv Neurol Disord 2022; 15:17562864221129380. [PMID: 36225969 PMCID: PMC9549180 DOI: 10.1177/17562864221129380] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2022] [Accepted: 09/09/2022] [Indexed: 11/05/2022] Open
Abstract
Background Previous studies found that Asians seemed to have higher risk of HT after thrombolysis than Caucasians due to its race differences in genetic polymorphism. Whether the model developed by Caucasians could predict risk of symptomatic intracerebral hemorrhage (sICH) in Asians was unknown. Objectives To develop a machine learning-based model for predicting sICH after stroke thrombolysis in Caucasians and externally validate it in an independent Han Chinese cohort. Design The derivation Caucasian sample included 1738 ischemic stroke (IS) patients from the Virtual International Stroke Trials Archive (VISTA) data set, and the external validation Han Chinese cohort included 296 IS patients who were treated with intravenous thrombolysis. Methods Twenty-eight variables were collected across both samples. According to their properties, we classified them into six distinct clusters (ie, demographic variables, medical history, previous medication, baseline blood biomarkers, neuroimaging markers on initial CT scan and clinical characteristics). A support vector machine (SVM) model, which consisted of data processing, model training, testing and a 10-fold cross-validation, was developed to predict the risk of sICH after stroke thrombolysis. The receiving operating characteristic (ROC) was used to assess the prediction performance of the SVM model. A domain contribution analysis was then performed to test which cluster had the highest influence on the performance of the model. Results In total, 85 (4.9%) patients developed sICH in the Caucasians, and 29 (9.8%) patients developed sICH in the Han Chinese cohort. Eight features including age, NIHSS score, SBP (systolic blood pressure), DBP (diastolic blood pressure), ALP (alkaline phosphatase), ALT (alanine transaminase), glucose, and creatine level were included in the final model, all of which were from demographic, clinical characteristics, and blood biomarkers clusters, respectively. The SVM model showed a good predictive performance in both Caucasians (AUC = 0.87) and Han Chinese patients (AUC = 0.74). Domain contribution analysis showed that inclusion/exclusion of clinical characteristic cluster (NIHSS score, SBP, and DBP), had the highest influence on the performance of predicting sICH in both Caucasian and Han Chinese cohorts. Conclusion The established SVM model is feasible for predicting the risk of sICH after thrombolysis quickly and efficiently in both Caucasian and Han Chinese cohort.
Collapse
Affiliation(s)
- Junfeng Liu
- Department of Neurology, West China Hospital,
Sichuan University, Chengdu, China
| | - Xinyue Chen
- CT Collaboration, Siemens Healthineers,
Chengdu, China
| | - Xiaonan Guo
- School of Information Science and Engineering,
Yanshan University, Qinhuangdao, China
| | - Renjie Xu
- Department of Respiratory Medicine, West China
Hospital, Sichuan University, Chengdu, China
| | - Yanan Wang
- Department of Neurology, West China Hospital,
Sichuan University, Chengdu, China
| | | |
Collapse
|
12
|
Soni M, Wijeratne T, Ackland DC. A risk score for prediction of symptomatic intracerebral haemorrhage following thrombolysis. Int J Med Inform 2021; 156:104586. [PMID: 34649112 DOI: 10.1016/j.ijmedinf.2021.104586] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2021] [Revised: 08/29/2021] [Accepted: 09/17/2021] [Indexed: 10/20/2022]
Abstract
BACKGROUND AND PURPOSE Intravenous recombinant tissue plasminogen activator (rt-PA) remains the only FDA approved pharmacological therapy for acute ischemic stroke (AIS), but this treatment is associated with symptomatic intracerebral haemorrhage (SICH). The aim of this study was to derive and validate an accurate measure of SICH risk in ischemic stroke patients treated with rt-PA using data readily available from patient clinical records. METHODS Demographics, physiological parameters, and clinical data were obtained from 1,270 ischemic stroke patients treated with thrombolysis at 20 hospitals. This included age, sex, weight, blood pressure, glucose levels, smoking preferences, and presence of previous clinical conditions. Using a bivariate analysis on a training dataset of 890 patients, SICH cases were compared against SICH-free patients and key risk factors associated with SICH were identified. Continuous variables were stratified using k-means clustering, and odds ratios computed for each of the categorical risk factors employed in the risk score. The SICH risk score, which was assessed using an independent validation dataset comprising 380 patients, was defined between 0 and 53, and stratified into 4 categories: very low risk (0-6), low risk (7-12), moderate risk (13-19), and high risk (>20). RESULTS Older age (age > 75 years), higher blood pressure, higher severity of stroke, pre-treatment antithrombotic and history of hypertension and hyperlipidaemia, were shown to be significant risk factors for SICH following rt-PA treatment (p < 0.05). A number of interaction effects with age produced greater overall SICH risk than that of individual variables alone, including age*weight, age*NIHSS, age*diastolic blood pressure, and age*hypertension. The SICH prediction tool demonstrated a C-statistic of 0.75 for continuous risk scoring (0-53) and 0.71 for stratified risk levels. CONCLUSION A novel, computationally efficient risk score utilising data readily available from patient clinical records was shown to predict SICH risk following thrombolysis treatment with high accuracy. This tool may be useful for pre-screening patients for SICH risk to reduce the morbidity and mortality associated with thrombolysis treatment.
Collapse
Affiliation(s)
- Mukesh Soni
- Department of Biomedical Engineering, The University of Melbourne, Australia
| | - Tissa Wijeratne
- Department of Medicine and Neurology, AIMSS, Melbourne Medical School, University of Melbourne and Western Health, Sunshine Hospital, St. Albans, Victoria, Australia; School of Psychology & Public Health, Department of Psychology & Counselling, La Trobe University, Bundoora, VIC, Australia; Department of Medicine, Faculty of Medicine, University of Rajarata, Saliyapura, Anuradhapura, Sri Lanka
| | - David C Ackland
- Department of Biomedical Engineering, The University of Melbourne, Australia.
| |
Collapse
|
13
|
Choi JM, Seo SY, Kim PJ, Kim YS, Lee SH, Sohn JH, Kim DK, Lee JJ, Kim C. Prediction of Hemorrhagic Transformation after Ischemic Stroke Using Machine Learning. J Pers Med 2021; 11:863. [PMID: 34575640 PMCID: PMC8470833 DOI: 10.3390/jpm11090863] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2021] [Revised: 08/25/2021] [Accepted: 08/28/2021] [Indexed: 12/27/2022] Open
Abstract
Hemorrhagic transformation (HT) is one of the leading causes of a poor prognostic marker after acute ischemic stroke (AIS). We compared the performances of the several machine learning (ML) algorithms to predict HT after AIS using only structured data. A total of 2028 patients with AIS, who were admitted within seven days of symptoms onset, were included in this analysis. HT was defined based on the criteria of the European Co-operative Acute Stroke Study-II trial. The whole dataset was randomly divided into a training and a test dataset with a 7:3 ratio. Binary logistic regression, support vector machine, extreme gradient boosting, and artificial neural network (ANN) algorithms were used to assess the performance of predicting the HT occurrence after AIS. Five-fold cross validation and a grid search technique were used to optimize the hyperparameters of each ML model, which had its performance measured by the area under the receiver operating characteristic (AUROC) curve. Among the included AIS patients, the mean age and number of male subjects were 69.6 years and 1183 (58.3%), respectively. HT was observed in 318 subjects (15.7%). There were no significant differences in corresponding variables between the training and test dataset. Among all the ML algorithms, the ANN algorithm showed the best performance in terms of predicting the occurrence of HT in our dataset (0.844). Feature scaling including standardization and normalization, and the resampling strategy showed no additional improvement of the ANN's performance. The ANN-based prediction of HT after AIS showed better performance than the conventional ML algorithms. Deep learning may be used to predict important outcomes for structured data-based prediction.
Collapse
Affiliation(s)
- Jeong-Myeong Choi
- Department of Convergence Software, Hallym University, Chuncheon 24252, Korea; (J.-M.C.); (S.-Y.S.); (Y.-S.K.)
| | - Soo-Young Seo
- Department of Convergence Software, Hallym University, Chuncheon 24252, Korea; (J.-M.C.); (S.-Y.S.); (Y.-S.K.)
| | - Pum-Jun Kim
- Institute of New Frontier Research Team, Hallym University College of Medicine, Chuncheon 24252, Korea; (P.-J.K.); (S.-H.L.); (J.-H.S.); (D.-K.K.); (J.-J.L.)
| | - Yu-Seop Kim
- Department of Convergence Software, Hallym University, Chuncheon 24252, Korea; (J.-M.C.); (S.-Y.S.); (Y.-S.K.)
| | - Sang-Hwa Lee
- Institute of New Frontier Research Team, Hallym University College of Medicine, Chuncheon 24252, Korea; (P.-J.K.); (S.-H.L.); (J.-H.S.); (D.-K.K.); (J.-J.L.)
- Department of Neurology, Chuncheon Sacred Heart Hospital, Chuncheon 24253, Korea
| | - Jong-Hee Sohn
- Institute of New Frontier Research Team, Hallym University College of Medicine, Chuncheon 24252, Korea; (P.-J.K.); (S.-H.L.); (J.-H.S.); (D.-K.K.); (J.-J.L.)
- Department of Neurology, Chuncheon Sacred Heart Hospital, Chuncheon 24253, Korea
| | - Dong-Kyu Kim
- Institute of New Frontier Research Team, Hallym University College of Medicine, Chuncheon 24252, Korea; (P.-J.K.); (S.-H.L.); (J.-H.S.); (D.-K.K.); (J.-J.L.)
- Department of Otorhinolaryngology and Head and Neck Surgery, Chuncheon Sacred Heart Hospital, Chuncheon 24253, Korea
| | - Jae-Jun Lee
- Institute of New Frontier Research Team, Hallym University College of Medicine, Chuncheon 24252, Korea; (P.-J.K.); (S.-H.L.); (J.-H.S.); (D.-K.K.); (J.-J.L.)
- Department of Anesthesiology and Pain Medicine, Chuncheon Sacred Heart Hospital, Chuncheon 24253, Korea
| | - Chulho Kim
- Institute of New Frontier Research Team, Hallym University College of Medicine, Chuncheon 24252, Korea; (P.-J.K.); (S.-H.L.); (J.-H.S.); (D.-K.K.); (J.-J.L.)
- Department of Neurology, Chuncheon Sacred Heart Hospital, Chuncheon 24253, Korea
| |
Collapse
|
14
|
Osama S, Zafar K, Sadiq MU. Predicting Clinical Outcome in Acute Ischemic Stroke Using Parallel Multi-parametric Feature Embedded Siamese Network. Diagnostics (Basel) 2020; 10:E858. [PMID: 33105609 PMCID: PMC7690444 DOI: 10.3390/diagnostics10110858] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2020] [Revised: 10/16/2020] [Accepted: 10/19/2020] [Indexed: 11/16/2022] Open
Abstract
Stroke is the second leading cause of death and disability worldwide, with ischemic stroke as the most common type. The preferred diagnostic procedure at the acute stage is the acquisition of multi-parametric magnetic resonance imaging (MRI). This type of imaging not only detects and locates the stroke lesion, but also provides the blood flow dynamics that helps clinicians in assessing the risks and benefits of reperfusion therapies. However, evaluating the outcome of these risky therapies beforehand is a complicated task due to the variability of lesion location, size, shape, and cerebral hemodynamics involved. Though the fully automated model for predicting treatment outcomes using multi-parametric imaging would be highly valuable in clinical settings, MRI datasets acquired at the acute stage are mostly scarce and suffer high class imbalance. In this paper, parallel multi-parametric feature embedded siamese network (PMFE-SN) is proposed that can learn with few samples and can handle skewness in multi-parametric MRI data. Moreover, five suitable evaluation metrics that are insensitive to imbalance are defined for this problem. The results show that PMFE-SN not only outperforms other state-of-the-art techniques in all these metrics but also can predict the class with a small number of samples, as well as the class with high number of samples. An accuracy of 0.67 on leave one cross out testing has been achieved with only two samples (minority class) for training and accuracy of 0.61 with the highest number of samples (majority class). In comparison, state-of-the-art using hand crafted features has 0 accuracy for minority class and 0.33 accuracy for majority class.
Collapse
Affiliation(s)
| | - Kashif Zafar
- Department of Computer Science, National University of Computing and Emerging Sciences, 852-B Milaad St, Block B Faisal Town, Lahore 54000, Pakistan; (S.O.); (M.U.S.)
| | | |
Collapse
|