1
|
Chen Z, Hu B, Liu X, Becker B, Eickhoff SB, Miao K, Gu X, Tang Y, Dai X, Li C, Leonov A, Xiao Z, Feng Z, Chen J, Chuan-Peng H. Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry. BMC Med 2023; 21:241. [PMID: 37400814 DOI: 10.1186/s12916-023-02941-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/10/2023] [Accepted: 06/13/2023] [Indexed: 07/05/2023] Open
Abstract
BACKGROUND The development of machine learning models for aiding in the diagnosis of mental disorder is recognized as a significant breakthrough in the field of psychiatry. However, clinical practice of such models remains a challenge, with poor generalizability being a major limitation. METHODS Here, we conducted a pre-registered meta-research assessment on neuroimaging-based models in the psychiatric literature, quantitatively examining global and regional sampling issues over recent decades, from a view that has been relatively underexplored. A total of 476 studies (n = 118,137) were included in the current assessment. Based on these findings, we built a comprehensive 5-star rating system to quantitatively evaluate the quality of existing machine learning models for psychiatric diagnoses. RESULTS A global sampling inequality in these models was revealed quantitatively (sampling Gini coefficient (G) = 0.81, p < .01), varying across different countries (regions) (e.g., China, G = 0.47; the USA, G = 0.58; Germany, G = 0.78; the UK, G = 0.87). Furthermore, the severity of this sampling inequality was significantly predicted by national economic levels (β = - 2.75, p < .001, R2adj = 0.40; r = - .84, 95% CI: - .41 to - .97), and was plausibly predictable for model performance, with higher sampling inequality for reporting higher classification accuracy. Further analyses showed that lack of independent testing (84.24% of models, 95% CI: 81.0-87.5%), improper cross-validation (51.68% of models, 95% CI: 47.2-56.2%), and poor technical transparency (87.8% of models, 95% CI: 84.9-90.8%)/availability (80.88% of models, 95% CI: 77.3-84.4%) are prevailing in current diagnostic classifiers despite improvements over time. Relating to these observations, model performances were found decreased in studies with independent cross-country sampling validations (all p < .001, BF10 > 15). In light of this, we proposed a purpose-built quantitative assessment checklist, which demonstrated that the overall ratings of these models increased by publication year but were negatively associated with model performance. CONCLUSIONS Together, improving sampling economic equality and hence the quality of machine learning models may be a crucial facet to plausibly translating neuroimaging-based diagnostic classifiers into clinical practice.
Collapse
Affiliation(s)
- Zhiyi Chen
- Experimental Research Center for Medical and Psychological Science (ERC-MPS), School of Psychology, Third Military Medical University, Chongqing, China.
- Faculty of Psychology, Southwest University, Chongqing, China.
| | - Bowen Hu
- Faculty of Psychology, Southwest University, Chongqing, China
| | - Xuerong Liu
- Experimental Research Center for Medical and Psychological Science (ERC-MPS), School of Psychology, Third Military Medical University, Chongqing, China
| | - Benjamin Becker
- The Center of Psychosomatic Medicine, Sichuan Provincial Center for Mental Health, Sichuan Provincial People's Hospital, Chengdu, China
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Laboratory for Neuroinformation, University of Electronic Science and Technology of China, Chengdu, China
| | - Simon B Eickhoff
- Institute of Systems Neuroscience, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Kuan Miao
- Experimental Research Center for Medical and Psychological Science (ERC-MPS), School of Psychology, Third Military Medical University, Chongqing, China
| | - Xingmei Gu
- Experimental Research Center for Medical and Psychological Science (ERC-MPS), School of Psychology, Third Military Medical University, Chongqing, China
| | - Yancheng Tang
- School of Business and Management, Shanghai International Studies University, Shanghai, China
| | - Xin Dai
- Faculty of Psychology, Southwest University, Chongqing, China
| | - Chao Li
- Department of Radiology, The Third Affiliated Hospital, Sun Yat-Sen University, Guangdong, China
| | - Artemiy Leonov
- School of Psychology, Clark University, Worcester, MA, USA
| | - Zhibing Xiao
- State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, China
| | - Zhengzhi Feng
- Experimental Research Center for Medical and Psychological Science (ERC-MPS), School of Psychology, Third Military Medical University, Chongqing, China
| | - Ji Chen
- Department of Psychology and Behavioral Sciences, Zhejiang University, Hangzhou, China.
- Department of Psychiatry, The Fourth Affiliated Hospital, Zhejiang University School of Medicine, Yiwu, Zhejiang, China.
| | - Hu Chuan-Peng
- School of Psychology, Nanjing Normal University, Nanjing, China
| |
Collapse
|
2
|
Chen Z, Liu X, Yang Q, Wang YJ, Miao K, Gong Z, Yu Y, Leonov A, Liu C, Feng Z, Chuan-Peng H. Evaluation of Risk of Bias in Neuroimaging-Based Artificial Intelligence Models for Psychiatric Diagnosis: A Systematic Review. JAMA Netw Open 2023; 6:e231671. [PMID: 36877519 PMCID: PMC9989906 DOI: 10.1001/jamanetworkopen.2023.1671] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 03/07/2023] Open
Abstract
IMPORTANCE Neuroimaging-based artificial intelligence (AI) diagnostic models have proliferated in psychiatry. However, their clinical applicability and reporting quality (ie, feasibility) for clinical practice have not been systematically evaluated. OBJECTIVE To systematically assess the risk of bias (ROB) and reporting quality of neuroimaging-based AI models for psychiatric diagnosis. EVIDENCE REVIEW PubMed was searched for peer-reviewed, full-length articles published between January 1, 1990, and March 16, 2022. Studies aimed at developing or validating neuroimaging-based AI models for clinical diagnosis of psychiatric disorders were included. Reference lists were further searched for suitable original studies. Data extraction followed the CHARMS (Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies) and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines. A closed-loop cross-sequential design was used for quality control. The PROBAST (Prediction Model Risk of Bias Assessment Tool) and modified CLEAR (Checklist for Evaluation of Image-Based Artificial Intelligence Reports) benchmarks were used to systematically evaluate ROB and reporting quality. FINDINGS A total of 517 studies presenting 555 AI models were included and evaluated. Of these models, 461 (83.1%; 95% CI, 80.0%-86.2%) were rated as having a high overall ROB based on the PROBAST. The ROB was particular high in the analysis domain, including inadequate sample size (398 of 555 models [71.7%; 95% CI, 68.0%-75.6%]), poor model performance examination (with 100% of models lacking calibration examination), and lack of handling data complexity (550 of 555 models [99.1%; 95% CI, 98.3%-99.9%]). None of the AI models was perceived to be applicable to clinical practices. Overall reporting completeness (ie, number of reported items/number of total items) for the AI models was 61.2% (95% CI, 60.6%-61.8%), and the completeness was poorest for the technical assessment domain with 39.9% (95% CI, 38.8%-41.1%). CONCLUSIONS AND RELEVANCE This systematic review found that the clinical applicability and feasibility of neuroimaging-based AI models for psychiatric diagnosis were challenged by a high ROB and poor reporting quality. Particularly in the analysis domain, ROB in AI diagnostic models should be addressed before clinical application.
Collapse
Affiliation(s)
- Zhiyi Chen
- School of Psychology, Third Military Medical University, Chongqing, China
- Experimental Research Center for Medical and Psychological Science, Third Military Medical University, Chongqing, China
| | - Xuerong Liu
- School of Psychology, Third Military Medical University, Chongqing, China
- Experimental Research Center for Medical and Psychological Science, Third Military Medical University, Chongqing, China
| | - Qingwu Yang
- Department of Neurology, Daping Hospital, Third Military Medical University, Chongqing, China
| | - Yan-Jiang Wang
- Department of Neurology, Daping Hospital, Third Military Medical University, Chongqing, China
| | - Kuan Miao
- School of Psychology, Third Military Medical University, Chongqing, China
- Experimental Research Center for Medical and Psychological Science, Third Military Medical University, Chongqing, China
| | - Zheng Gong
- School of Psychology, Third Military Medical University, Chongqing, China
- Experimental Research Center for Medical and Psychological Science, Third Military Medical University, Chongqing, China
| | - Yang Yu
- School of Psychology, Third Military Medical University, Chongqing, China
| | - Artemiy Leonov
- Department of Psychology, Clark University, Worcester, Massachusetts
| | - Chunlei Liu
- School of Psychology, Qufu Normal University, Qufu, China
| | - Zhengzhi Feng
- School of Psychology, Third Military Medical University, Chongqing, China
- Experimental Research Center for Medical and Psychological Science, Third Military Medical University, Chongqing, China
| | - Hu Chuan-Peng
- School of Psychology, Nanjing Normal University, Nanjing, China
| |
Collapse
|
3
|
Khojaste-Sarakhsi M, Haghighi SS, Ghomi SF, Marchiori E. Deep learning for Alzheimer's disease diagnosis: A survey. Artif Intell Med 2022; 130:102332. [DOI: 10.1016/j.artmed.2022.102332] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2020] [Revised: 04/29/2022] [Accepted: 05/30/2022] [Indexed: 11/28/2022]
|
4
|
Going deep into schizophrenia with artificial intelligence. Schizophr Res 2022; 245:122-140. [PMID: 34103242 DOI: 10.1016/j.schres.2021.05.018] [Citation(s) in RCA: 19] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/14/2020] [Revised: 05/24/2021] [Accepted: 05/27/2021] [Indexed: 12/30/2022]
Abstract
Despite years of research, the mechanisms governing the onset, relapse, symptomatology, and treatment of schizophrenia (SZ) remain elusive. The lack of appropriate analytic tools to deal with the heterogeneity and complexity of SZ may be one of the reasons behind this situation. Deep learning, a subfield of artificial intelligence (AI) inspired by the nervous system, has recently provided an accessible way of modeling and analyzing complex, high-dimensional, nonlinear systems. The unprecedented accuracy of deep learning algorithms in classification and prediction tasks has revolutionized a wide range of scientific fields and is rapidly permeating SZ research. Deep learning has the potential of becoming a valuable aid for clinicians in the prediction, diagnosis, and treatment of SZ, especially in combination with principles from Bayesian statistics. Furthermore, deep learning could become a powerful tool for uncovering the mechanisms underlying SZ thanks to a growing number of techniques designed for improving model interpretability and causal reasoning. The purpose of this article is to introduce SZ researchers to the field of deep learning and review its latest applications in SZ research. In general, existing studies have yielded impressive results in classification and outcome prediction tasks. However, methodological concerns related to the assessment of model performance in several studies, the widespread use of small training datasets, and the little clinical value of some models suggest that some of these results should be taken with caution.
Collapse
|
5
|
Barros C, Roach B, Ford JM, Pinheiro AP, Silva CA. From Sound Perception to Automatic Detection of Schizophrenia: An EEG-Based Deep Learning Approach. Front Psychiatry 2022; 12:813460. [PMID: 35250651 PMCID: PMC8892210 DOI: 10.3389/fpsyt.2021.813460] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/11/2021] [Accepted: 12/31/2021] [Indexed: 12/27/2022] Open
Abstract
Deep learning techniques have been applied to electroencephalogram (EEG) signals, with promising applications in the field of psychiatry. Schizophrenia is one of the most disabling neuropsychiatric disorders, often characterized by the presence of auditory hallucinations. Auditory processing impairments have been studied using EEG-derived event-related potentials and have been associated with clinical symptoms and cognitive dysfunction in schizophrenia. Due to consistent changes in the amplitude of ERP components, such as the auditory N100, some have been proposed as biomarkers of schizophrenia. In this paper, we examine altered patterns in electrical brain activity during auditory processing and their potential to discriminate schizophrenia and healthy subjects. Using deep convolutional neural networks, we propose an architecture to perform the classification based on multi-channels auditory-related EEG single-trials, recorded during a passive listening task. We analyzed the effect of the number of electrodes used, as well as the laterality and distribution of the electrical activity over the scalp. Results show that the proposed model is able to classify schizophrenia and healthy subjects with an average accuracy of 78% using only 5 midline channels (Fz, FCz, Cz, CPz, and Pz). The present study shows the potential of deep learning methods in the study of impaired auditory processing in schizophrenia with implications for diagnosis. The proposed design can provide a base model for future developments in schizophrenia research.
Collapse
Affiliation(s)
- Carla Barros
- Psychological Neurosciences Lab, Psychology Research Center (CIPsi), School of Psychology, University of Minho, Braga, Portugal
| | - Brian Roach
- Psychiatry Service, San Francisco Veteran Affairs Medical Center (VAMC), San Francisco, CA, United States
- Department of Psychiatry, University of California, San Francisco, San Francisco, CA, United States
| | - Judith M. Ford
- Psychiatry Service, San Francisco Veteran Affairs Medical Center (VAMC), San Francisco, CA, United States
- Department of Psychiatry, University of California, San Francisco, San Francisco, CA, United States
| | - Ana P. Pinheiro
- Psychological Neurosciences Lab, Psychology Research Center (CIPsi), School of Psychology, University of Minho, Braga, Portugal
- Research Center for Psychological Science (CICPSI), Faculdade de Psicologia, Universidade de Lisboa, Lisbon, Portugal
| | - Carlos A. Silva
- Center for MicroElectromechanical Systems (CMEMS-UMinho), University of Minho, Guimarães, Portugal
- LABBELS - Associate Laboratory, Guimarães, Portugal
| |
Collapse
|
6
|
A multiscale siamese convolutional neural network with cross-channel fusion for motor imagery decoding. J Neurosci Methods 2022; 367:109426. [PMID: 34902364 DOI: 10.1016/j.jneumeth.2021.109426] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2021] [Revised: 10/28/2021] [Accepted: 11/22/2021] [Indexed: 11/22/2022]
Abstract
BACKGROUND Recently, convolutional neural networks (CNN) are widely applied in motor imagery electroencephalography (MI-EEG) signal classification tasks. However, a simple CNN framework is challenging to satisfy the complex MI-EEG signal decoding. NEW METHOD In this study, we propose a multiscale Siamese convolutional neural network with cross-channel fusion (MSCCF-Net) for MI-EEG classification tasks. The proposed network consists of three parts: Siamese cross-channel fusion streams, similarity module and classification module. Each Siamese cross-channel fusion stream contains multiple branches, and each branch is supplemented by cross-channel fusion modules to improve multiscale temporal feature representation capability. The similarity module is adopted to measure the feature similarity between multiple branches. At the same time, the classification module provides a strong constraint to classify the features from all Siamese cross-channel fusion streams. The combination of the similarity module and classification module constitutes a new joint training strategy to further optimize the network performance. RESULTS The experiment is conducted on the public BCI Competition IV 2a and 2b datasets, and the results show that the proposed network achieves an average accuracy of 87.36% and 87.33%, respectively. COMPARISON WITH EXISTING METHODS AND CONCLUSIONS The proposed network adopts cross-channel fusion to learn multiscale temporal characteristics and joint training strategy to optimize the training process. Therefore, the performance outperforms other state-of-the-art MI-EEG signal classification methods.
Collapse
|
7
|
Medical Applications of Artificial Intelligence (Legal Aspects and Future Prospects). LAWS 2021. [DOI: 10.3390/laws11010003] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
Abstract
Background: Cutting-edge digital technologies are being actively introduced into healthcare. The recent successful efforts of artificial intelligence in diagnosing, predicting and studying diseases, as well as in surgical assisting demonstrate its high efficiency. The AI’s ability to promptly take decisions and learn independently has motivated large corporations to focus on its development and gradual introduction into everyday life. Legal aspects of medical activities are of particular importance, yet the legal regulation of AI’s performance in healthcare is still in its infancy. The state is to a considerable extent responsible for the formation of a legal regime that would meet the needs of modern society (digital society). Objective: This study aims to determine the possible modes of AI’s functioning, to identify the participants in medical-legal relations, to define the legal personality of AI and circumscribe the scope of its competencies. Of importance is the issue of determining the grounds for imposing legal liability on persons responsible for the performance of an AI system. Results: The present study identifies the prospects for a legal assessment of AI applications in medicine. The article reviews the sources of legal regulation of AI, including the unique sources of law sanctioned by the state. Particular focus is placed on medical-legal customs and medical practices. Conclusions: The presented analysis has allowed formulating the approaches to the legal regulation of AI in healthcare.
Collapse
|
8
|
Baygin M, Yaman O, Tuncer T, Dogan S, Barua PD, Acharya UR. Automated accurate schizophrenia detection system using Collatz pattern technique with EEG signals. Biomed Signal Process Control 2021. [DOI: 10.1016/j.bspc.2021.102936] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/20/2022]
|
9
|
Gonçalves DM, Henriques R, Costa RS. Predicting Postoperative Complications in Cancer Patients: A Survey Bridging Classical and Machine Learning Contributions to Postsurgical Risk Analysis. Cancers (Basel) 2021; 13:cancers13133217. [PMID: 34203189 PMCID: PMC8269422 DOI: 10.3390/cancers13133217] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2021] [Revised: 06/04/2021] [Accepted: 06/22/2021] [Indexed: 02/05/2023] Open
Abstract
Simple Summary Structured survey on the predictive analysis of postoperative complications in oncology, bridging classic risk scores with machine learning advances, and further establishing principles to guide the design of cohort studies and the predictive modeling of postsurgical risks. Abstract Postoperative complications can impose a significant burden, increasing morbidity, mortality, and the in-hospital length of stay. Today, the number of studies available on the prognostication of postsurgical complications in cancer patients is growing and has already created a considerable set of dispersed contributions. This work provides a comprehensive survey on postoperative risk analysis, integrating principles from classic risk scores and machine-learning approaches within a coherent frame. A qualitative comparison is offered, taking into consideration the available cohort data and the targeted postsurgical outcomes of morbidity (such as the occurrence, nature or severity of postsurgical complications and hospitalization needs) and mortality. This work further establishes a taxonomy to assess the adequacy of cohort studies and guide the development and assessment of new learning approaches for the study and prediction of postoperative complications.
Collapse
Affiliation(s)
- Daniel M. Gonçalves
- IDMEC, Instituto Superior Técnico, Universidade de Lisboa, Av. Rovisco Pais 1, 1049-001 Lisboa, Portugal; (D.M.G.); (R.S.C.)
- INESC-ID, Lisboa Portugal and Instituto Superior Técnico, Universidade de Lisboa, R. Alves Redol 9, 1000-029 Lisboa, Portugal
| | - Rui Henriques
- INESC-ID, Lisboa Portugal and Instituto Superior Técnico, Universidade de Lisboa, R. Alves Redol 9, 1000-029 Lisboa, Portugal
- Correspondence: ; Tel.: +351-21-310-0300
| | - Rafael S. Costa
- IDMEC, Instituto Superior Técnico, Universidade de Lisboa, Av. Rovisco Pais 1, 1049-001 Lisboa, Portugal; (D.M.G.); (R.S.C.)
- LAQV-REQUIMTE, NOVA School of Science and Technology, Campus Caparica, Universidade NOVA de Lisboa, 2829-516 Caparica, Portugal
| |
Collapse
|
10
|
Diagnosis and prognosis of mental disorders by means of EEG and deep learning: a systematic mapping study. Artif Intell Rev 2021. [DOI: 10.1007/s10462-021-09986-y] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/16/2023]
|