1
|
Rossi L, Fiorentino MC, Mancini A, Paolanti M, Rosati R, Zingaretti P. Generalizability and robustness evaluation of attribute-based zero-shot learning. Neural Netw 2024; 175:106278. [PMID: 38581809 DOI: 10.1016/j.neunet.2024.106278] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2023] [Revised: 02/15/2024] [Accepted: 03/26/2024] [Indexed: 04/08/2024]
Abstract
In the field of deep learning, large quantities of data are typically required to effectively train models. This challenge has given rise to techniques like zero-shot learning (ZSL), which trains models on a set of "seen" classes and evaluates them on a set of "unseen" classes. Although ZSL has shown considerable potential, particularly with the employment of generative methods, its generalizability to real-world scenarios remains uncertain. The hypothesis of this work is that the performance of ZSL models is systematically influenced by the chosen "splits"; in particular, the statistical properties of the classes and attributes used in training. In this paper, we test this hypothesis by introducing the concepts of generalizability and robustness in attribute-based ZSL and carry out a variety of experiments to stress-test ZSL models against different splits. Our aim is to lay the groundwork for future research on ZSL models' generalizability, robustness, and practical applications. We evaluate the accuracy of state-of-the-art models on benchmark datasets and identify consistent trends in generalizability and robustness. We analyze how these properties vary based on the dataset type, differentiating between coarse- and fine-grained datasets, and our findings indicate significant room for improvement in both generalizability and robustness. Furthermore, our results demonstrate the effectiveness of dimensionality reduction techniques in improving the performance of state-of-the-art models in fine-grained datasets.
Collapse
Affiliation(s)
- Luca Rossi
- Dipartimento di Ingegneria dell'Informazione, Università Politecnica delle Marche, Via Brecce Bianche 12, 60131, Ancona, Italy.
| | - Maria Chiara Fiorentino
- Dipartimento di Ingegneria dell'Informazione, Università Politecnica delle Marche, Via Brecce Bianche 12, 60131, Ancona, Italy.
| | - Adriano Mancini
- Dipartimento di Ingegneria dell'Informazione, Università Politecnica delle Marche, Via Brecce Bianche 12, 60131, Ancona, Italy.
| | - Marina Paolanti
- Dipartimento di Scienze politiche, della Comunicazione e delle Relazioni Internazionali, Università di Macerata, 62100, Macerata, Italy.
| | - Riccardo Rosati
- Dipartimento di Ingegneria dell'Informazione, Università Politecnica delle Marche, Via Brecce Bianche 12, 60131, Ancona, Italy.
| | - Primo Zingaretti
- Dipartimento di Ingegneria dell'Informazione, Università Politecnica delle Marche, Via Brecce Bianche 12, 60131, Ancona, Italy.
| |
Collapse
|
2
|
Stavropoulos A, Crone DL, Grossmann I. Shadows of wisdom: Classifying meta-cognitive and morally grounded narrative content via large language models. Behav Res Methods 2024:10.3758/s13428-024-02441-0. [PMID: 38811519 DOI: 10.3758/s13428-024-02441-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/14/2024] [Indexed: 05/31/2024]
Abstract
We investigated large language models' (LLMs) efficacy in classifying complex psychological constructs like intellectual humility, perspective-taking, open-mindedness, and search for a compromise in narratives of 347 Canadian and American adults reflecting on a workplace conflict. Using state-of-the-art models like GPT-4 across few-shot and zero-shot paradigms and RoB-ELoC (RoBERTa -fine-tuned-on-Emotion-with-Logistic-Regression-Classifier), we compared their performance with expert human coders. Results showed robust classification by LLMs, with over 80% agreement and F1 scores above 0.85, and high human-model reliability (Cohen's κ Md across top models = .80). RoB-ELoC and few-shot GPT-4 were standout classifiers, although somewhat less effective in categorizing intellectual humility. We offer example workflows for easy integration into research. Our proof-of-concept findings indicate the viability of both open-source and commercial LLMs in automating the coding of complex constructs, potentially transforming social science research.
Collapse
Affiliation(s)
| | | | - Igor Grossmann
- Department of Psychology, University of Waterloo, Waterloo, N2L 3G1, Canada.
| |
Collapse
|
3
|
Kaufmann B, Busby D, Das CK, Tillu N, Menon M, Tewari AK, Gorin MA. Validation of a Zero-shot Learning Natural Language Processing Tool to Facilitate Data Abstraction for Urologic Research. Eur Urol Focus 2024; 10:279-287. [PMID: 38278710 DOI: 10.1016/j.euf.2024.01.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2023] [Revised: 12/18/2023] [Accepted: 01/15/2024] [Indexed: 01/28/2024]
Abstract
BACKGROUND Urologic research often requires data abstraction from unstructured text contained within the electronic health record. A number of natural language processing (NLP) tools have been developed to aid with this time-consuming task; however, the generalizability of these tools is typically limited by the need for task-specific training. OBJECTIVE To describe the development and validation of a zero-shot learning NLP tool to facilitate data abstraction from unstructured text for use in downstream urologic research. DESIGN, SETTING, AND PARTICIPANTS An NLP tool based on the GPT-3.5 model from OpenAI was developed and compared with three physicians for time to task completion and accuracy for abstracting 14 unique variables from a set of 199 deidentified radical prostatectomy pathology reports. The reports were processed in vectorized and scanned formats to establish the impact of optical character recognition on data abstraction. INTERVENTION A zero-shot learning NLP tool for data abstraction. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS The tool was compared with the human abstractors in terms of superiority for data abstraction speed and noninferiority for accuracy. RESULTS AND LIMITATIONS The human abstractors required a median (interquartile range) of 93 s (72-122 s) per report for data abstraction, whereas the software required a median of 12 s (10-15 s) for the vectorized reports and 15 s (13-17 s) for the scanned reports (p < 0.001 for all paired comparisons). The accuracies of the three human abstractors were 94.7% (95% confidence interval [CI], 93.8-95.5%), 97.8% (95% CI, 97.2-98.3%), and 96.4% (95% CI, 95.6-97%) for the combined set of 2786 data points. The tool had accuracy of 94.2% (95% CI, 93.3-94.9%) for the vectorized reports and was noninferior to the human abstractors at a margin of -10% (α = 0.025). The tool had slightly lower accuracy of 88.7% (95% CI 87.5-89.9%) for the scanned reports, making it noninferior to two of three human abstractors. CONCLUSIONS The developed zero-shot learning NLP tool offers urologic researchers a highly generalizable and accurate method for data abstraction from unstructured text. An open access version of the tool is available for immediate use by the urologic community. PATIENT SUMMARY In this report, we describe the design and validation of an artificial intelligence tool for abstracting discrete data from unstructured notes contained within the electronic medical record. This freely available tool, which is based on the GPT-3.5 technology from OpenAI, is intended to facilitate research and scientific discovery by the urologic community.
Collapse
Affiliation(s)
- Basil Kaufmann
- Milton and Carroll Petrie Department of Urology, Icahn School of Medicine at Mount Sinai, New York, NY, USA; Department of Urology, University Hospital Zurich, University of Zurich, Zurich, Switzerland.
| | - Dallin Busby
- Milton and Carroll Petrie Department of Urology, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Chandan Krushna Das
- Milton and Carroll Petrie Department of Urology, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Neeraja Tillu
- Milton and Carroll Petrie Department of Urology, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Mani Menon
- Milton and Carroll Petrie Department of Urology, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Ashutosh K Tewari
- Milton and Carroll Petrie Department of Urology, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Michael A Gorin
- Milton and Carroll Petrie Department of Urology, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| |
Collapse
|
4
|
Zhang H, Meng X, Cao W, Liu Y, Ming Z, Yang J. Graph embedding based multi-label Zero-shot Learning. Neural Netw 2023; 167:129-140. [PMID: 37657252 DOI: 10.1016/j.neunet.2023.08.023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2023] [Revised: 08/10/2023] [Accepted: 08/13/2023] [Indexed: 09/03/2023]
Abstract
Multi-label Zero-shot Learning (ZSL) is more reasonable and realistic than standard single-label ZSL because several objects can co-exist in a natural image in real scenarios. Intra-class feature entanglement is a significant factor influencing the alignment of visual and semantic features, resulting in the model's inability to recognize unseen samples comprehensively and completely. We observe that existing multi-label ZSL methods place a greater emphasis on attention-based refinement and decoupling of visual features, while ignoring the relationship between label semantics. Relying on label correlations to solve multi-label ZSL tasks has not been deeply studied. In this paper, we make full use of the co-occurrence relationship between category labels and build a directed weighted semantic graph based on statistics and prior knowledge, in which node features represent category semantics and weighted edges represent conditional probabilities of label co-occurrence. To guide the targeted extraction of visual features, node features and edge set weights are simultaneously updated and refined, and embedded into the visual feature extraction network from a global and local perspective. The proposed method's effectiveness was demonstrated by simulation results on two challenging multi-label ZSL benchmarks: NUS-WIDE and Open Images. In comparison to state-of-the-art models, our model achieves an absolute gain of 2.4% mAP on NUS-WIDE and 2.1% mAP on Open Images respectively.
Collapse
Affiliation(s)
- Haigang Zhang
- Institute of Applied Artificial Intelligence of the Guangdong-Hong Kong-Macao Greater Bay Area, Shenzhen Polytechnic, Shenzhen, 518055, China
| | - Xianglong Meng
- Institute of Applied Artificial Intelligence of the Guangdong-Hong Kong-Macao Greater Bay Area, Shenzhen Polytechnic, Shenzhen, 518055, China
| | - Weipeng Cao
- Guangdong Laboratory of Artificial Intelligence and Digital Economy (Shenzhen), Shenzhen, 518107, China.
| | - Ye Liu
- Guangdong Laboratory of Artificial Intelligence and Digital Economy (Shenzhen), Shenzhen, 518107, China
| | - Zhong Ming
- Guangdong Laboratory of Artificial Intelligence and Digital Economy (Shenzhen), Shenzhen, 518107, China
| | - Jinfeng Yang
- Institute of Applied Artificial Intelligence of the Guangdong-Hong Kong-Macao Greater Bay Area, Shenzhen Polytechnic, Shenzhen, 518055, China
| |
Collapse
|
5
|
Workman TE, Goulet JL, Brandt CA, Warren AR, Eleazer J, Skanderson M, Lindemann L, Blosnich JR, O'Leary J, Zeng‐Treitler Q. Identifying suicide documentation in clinical notes through zero-shot learning. Health Sci Rep 2023; 6:e1526. [PMID: 37706016 PMCID: PMC10495736 DOI: 10.1002/hsr2.1526] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2023] [Revised: 08/08/2023] [Accepted: 08/11/2023] [Indexed: 09/15/2023] Open
Abstract
Background and Aims In deep learning, a major difficulty in identifying suicidality and its risk factors in clinical notes is the lack of training samples given the small number of true positive instances among the number of patients screened. This paper describes a novel methodology that identifies suicidality in clinical notes by addressing this data sparsity issue through zero-shot learning. Our general aim was to develop a tool that leveraged zero-shot learning to effectively identify suicidality documentation in all types of clinical notes. Methods US Veterans Affairs clinical notes served as data. The training data set label was determined using diagnostic codes of suicide attempt and self-harm. We used a base string associated with the target label of suicidality to provide auxiliary information by narrowing the positive training cases to those containing the base string. We trained a deep neural network by mapping the training documents' contents to a semantic space. For comparison, we trained another deep neural network using the identical training data set labels, and bag-of-words features. Results The zero-shot learning model outperformed the baseline model in terms of area under the curve, sensitivity, specificity, and positive predictive value at multiple probability thresholds. In applying a 0.90 probability threshold, the methodology identified notes documenting suicidality but not associated with a relevant ICD-10-CM code, with 94% accuracy. Conclusion This method can effectively identify suicidality without manual annotation.
Collapse
Affiliation(s)
- Terri Elizabeth Workman
- Biomedical Informatics CenterThe George Washington UniversityWashingtonDistrict of ColumbiaUSA
- VA Medical CenterWashingtonDistrict of ColumbiaUSA
| | - Joseph L. Goulet
- Department of Emergency MedicineYale School of MedicineNew HavenConnecticutUSA
- VA Connecticut Healthcare SystemWest HavenConnecticutUSA
| | - Cynthia A. Brandt
- Department of Emergency MedicineYale School of MedicineNew HavenConnecticutUSA
- VA Connecticut Healthcare SystemWest HavenConnecticutUSA
| | - Allison R. Warren
- PRIME Center, VA Connecticut Healthcare SystemWest HavenConnecticutUSA
| | - Jacob Eleazer
- PRIME Center, VA Connecticut Healthcare SystemWest HavenConnecticutUSA
| | | | - Luke Lindemann
- VA Connecticut Healthcare SystemWest HavenConnecticutUSA
| | - John R. Blosnich
- Suzanne Dworak‐Peck School of Social WorkUniversity of Southern CaliforniaLos AngelesCaliforniaUSA
| | - John O'Leary
- VA Connecticut Healthcare SystemWest HavenConnecticutUSA
- Department of Internal MedicineYale School of MedicineWest HavenConnecticutUSA
| | - Qing Zeng‐Treitler
- Biomedical Informatics CenterThe George Washington UniversityWashingtonDistrict of ColumbiaUSA
- VA Medical CenterWashingtonDistrict of ColumbiaUSA
| |
Collapse
|
6
|
Yu B, Xie C, Tang P, Duan H. Multi-view graph representation with similarity diffusion for general zero-shot learning. Neural Netw 2023; 166:38-50. [PMID: 37480768 DOI: 10.1016/j.neunet.2023.06.045] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2022] [Revised: 06/21/2023] [Accepted: 06/30/2023] [Indexed: 07/24/2023]
Abstract
Zero-shot learning (ZSL) aims to predict unseen classes without using samples of these classes in model training. The ZSL has been widely used in many knowledge-based models and applications to predict various parameters, including categories, subjects, and anomalies, in different domains. Nonetheless, most existing ZSL methods require the pre-defined semantics or attributes of particular data environments. Therefore, these methods are difficult to be applied to general data environments, such as ImageNet and other real-world datasets and applications. Recent research has tried to use open knowledge to enhance the ZSL methods to adapt it to an open data environment. However, the performance of these methods is relatively low, namely the accuracy is normally below 10%, which is due to the inadequate semantics that can be used from open knowledge. Moreover, the latest methods suffer from a significant "semantic gap" problem between the generated features of unseen classes and the real features of seen classes. To this end, this paper proposes a multi-view graph representation with a similarity diffusion model, applying the ZSL tasks to general data environments. This model applies a multi-view graph to enhance the semantics fully and proposes an innovative diffusion method to augment the graph representation. In addition, a feature diffusion method is proposed to augment the multi-view graph representation and bridge the semantic gap to realize zero-shot predicting. The results of numerous experiments in general data environments and on benchmark datasets show that the proposed method can achieve new state-of-the-art results in the field of general zero-shot learning. Furthermore, seven ablation studies analyze the effects of the settings and different modules of the proposed method on its performance in detail and prove the effectiveness of each module.
Collapse
Affiliation(s)
- Beibei Yu
- School of Software, Yunnan University, Kunming, 650500, China.
| | - Cheng Xie
- School of Software, Yunnan University, Kunming, 650500, China.
| | - Peng Tang
- School of Software, Yunnan University, Kunming, 650500, China.
| | - Haoran Duan
- School of Software, Yunnan University, Kunming, 650500, China.
| |
Collapse
|
7
|
Wang Y, Feng L, Song X, Xu D, Zhai Y. Zero-Shot Image Classification Method Based on Attention Mechanism and Semantic Information Fusion. SENSORS (BASEL, SWITZERLAND) 2023; 23:2311. [PMID: 36850908 PMCID: PMC9966441 DOI: 10.3390/s23042311] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/04/2023] [Revised: 02/15/2023] [Accepted: 02/16/2023] [Indexed: 06/18/2023]
Abstract
The zero-shot image classification (ZSIC) is designed to solve the classification problem when the sample is very small, or the category is missing. A common method is to use attribute or word vectors as a priori category features (auxiliary information) and complete the domain transfer from training of seen classes to recognition of unseen classes by building a mapping between image features and a priori category features. However, feature extraction of the whole image lacks discrimination, and the amount of information of single attribute features or word vector features of categories is insufficient, which makes the matching degree between image features and prior class features not high and affects the accuracy of the ZSIC model. To this end, a spatial attention mechanism is designed, and an image feature extraction module based on this attention mechanism is constructed to screen critical features with discrimination. A semantic information fusion method based on matrix decomposition is proposed, which first decomposes the attribute features and then fuses them with the extracted word vector features of a dataset to achieve information expansion. Through the above two improvement measures, the classification accuracy of the ZSIC model for unseen images is improved. The experimental results on public datasets verify the effect and superiority of the proposed methods.
Collapse
Affiliation(s)
- Yaru Wang
- Department of Automation, North China Electric Power University, Baoding 071003, China
| | - Lilong Feng
- Department of Automation, North China Electric Power University, Baoding 071003, China
| | - Xiaoke Song
- Department of Automation, North China Electric Power University, Baoding 071003, China
| | - Dawei Xu
- Department of Automation, North China Electric Power University, Baoding 071003, China
- State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Yongjie Zhai
- Department of Automation, North China Electric Power University, Baoding 071003, China
| |
Collapse
|
8
|
Borowiec ML, Dikow RB, Frandsen PB, McKeeken A, Valentini G, White AE. Deep learning as a tool for ecology and evolution. Methods Ecol Evol 2022. [DOI: 10.1111/2041-210x.13901] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Affiliation(s)
- Marek L. Borowiec
- Entomology, Plant Pathology and Nematology University of Idaho Moscow ID USA
- Institute for Bioinformatics and Evolutionary Studies (IBEST) University of Idaho Moscow ID USA
| | - Rebecca B. Dikow
- Data Science Lab, Office of the Chief Information Officer Smithsonian Institution Washington DC USA
| | - Paul B. Frandsen
- Data Science Lab, Office of the Chief Information Officer Smithsonian Institution Washington DC USA
- Department of Plant and Wildlife Sciences Brigham Young University Provo UT USA
| | - Alexander McKeeken
- Entomology, Plant Pathology and Nematology University of Idaho Moscow ID USA
| | | | - Alexander E. White
- Data Science Lab, Office of the Chief Information Officer Smithsonian Institution Washington DC USA
- Department of Botany, National Museum of Natural History Smithsonian Institution Washington DC USA
| |
Collapse
|
9
|
Kong D, Li X, Wang S, Li J, Yin B. Learning visual-and-semantic knowledge embedding for zero-shot image classification. APPL INTELL 2022. [DOI: 10.1007/s10489-022-03443-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
|
10
|
Meta-DZSL: a meta-dictionary learning based approach to zero-shot recognition. APPL INTELL 2022. [DOI: 10.1007/s10489-022-03257-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
11
|
Teng Y, Song C, Wu B. Toward jointly understanding social relationships and characters from videos. APPL INTELL 2022. [DOI: 10.1007/s10489-021-02738-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
12
|
Zahoora U, Rajarajan M, Pan Z, Khan A. Zero-day Ransomware Attack Detection using Deep Contractive Autoencoder and Voting based Ensemble Classifier. APPL INTELL 2022. [DOI: 10.1007/s10489-022-03244-6] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/17/2023]
|
13
|
Li X, Gu J, Sun X, Li J, Tang S. Parameter identification of robot manipulators with unknown payloads using an improved chaotic sparrow search algorithm. APPL INTELL 2022. [DOI: 10.1007/s10489-021-02972-5] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/21/2023]
|